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SUMMARY 

The  objective  of  this  research  was  to  develop  a 
simple  methodology  to  test  for  learning  using  a  small  sample 
size  and  to  develop  a  procedure  for  measuring  the  rate  of 
learning  at  any  particular  trial.  For  this  research  the  time 
between  trials  was  considered  insignificant  in  affecting 
previously  gained  knowledge  and  the  error  between  any 
observation  and  its  expected  value,  z^,  is  assumed  to  be 
NID  (0,oc2). 

Assuming  learning  can  be  described  by  a  performance 
curve  of  the  form  z  =  1-at  b  two  linear  methods  and  one  non¬ 
linear  method  were  developed  to  test  for  learning  by  examining 
the  rate  of  learning  over  several  trials.  Since  the  curve 
is  monotonically  increasing  a  positive  slope  will  be 
interpreted  as  learning  and  a  zero  slope  will  correspond  to 
no  learning  occurring.  The  -linear  procedures  are  based  on 
testing  the  average  rate  of  learning  that  occurs  over  several 

trials.  Several  methods  for  estimating  the  average  rate  of 

2 

learning  and  the  variance  of  the  observations,  a£  ,  were 
investigated.  The  best  method  for  estimating  the  average 
rate  of  learning,  based  on  the  minimum  variance  of  the  estimate, 
was  the  linear  least  squares  regression,  LLSR  method,  and  the 
best  estimator  of  a  ,  which  resulted  in  the  most  powerful 
test,  was  computed  using  the  first  differences  of  the 
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observations . 

In  the  nonlinear  method,  estimates  for  a  and  the 

e 

parameters  "a"  and  "b"  are  obtained  and  a  test  on  the  degree 
of  nonlinearity  of  the  function  is  conducted  using  Beales 
measure  of  nonlinearity.  If  the  degree  of  nonlinearity  is 
small  enough  then  the  confidence  interval  for  the  slope  at 
any  trial  can  be  evaluated  by  using  linear  theory  approximations. 

In  a  comparison  of  the  two  procedures,  the  linear  methods 
were  more  powerful  tests,  however,  the  nonlinear  method  was 
able  to  provide  information  on  the  rate  of  learning  at  each 
trial  when  the  nonlinearity  conditions  were  satisfied  and 
significant  learning  was  detected.  The  more  powerful 
linear  test  procedure  was  the  LLSR  method,  which  can 
detect  an  average  rate  of  learning  over  15  trials  of  .01 
at  an  a  =  .05  level  95%  of  the  time  when  the  standard 
deviation  is  a£  <  .05. 
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CHAPTER  I 

INTRODUCTION 

If  a  comparison  of  two  or  more  systems  is  to  be 
meaningful  it  seems  reasonable  that  each  system  should  be 
operating  at  its  full  potential.  When  the  human  factor 
enters  into  the  operation  of  a  system,  the  system's  full 
potential  is  not  realized  until  the  operator(s)  are  "fully 
learned."  To  determine  if  the  operators  are  fully  learned, 
one  must  be  able  to  measure  the  rate  of  learning  that  is 
occurring  over  successive  periods  of  operating  the  system. 
Assuming  learning  can  be  described  by  a  monotonic  function, 
a  fully  learned  status  would  correspond  to  a  zero  rate  of 
learning . 

When  the  crew  operating  one  system  is  fully  learned 
while  the  crew  operating  a  second  system  is  not,  a  negative 
bias  could  be  introduced  into  the  test  results  of  the  second 
system,  rendering  an  unfair  evaluation.  This  bias  which  is 
the  result  of  a  difference  in  the  proficiency  levels  of 
respective  crews  on  competing  systems  is  a  major  concern  to 
the  U.  S.  Army  Operational  Test  and  Evaluation  Agency  (OTEA) 
during  operational  testing  and  evaluation  of  contracted 
equipment . 
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Background 

This  study  was  prompted  by  the  desire  of  the  U.  S. 
Army  Operational  Test  and  Evaluation  Agency  (OTEA)  deter¬ 
mine  if  a  crew  or  unit  is  fuily  learned  on  the  operation  of 
the  system  being  evaluated. 

The  purpose  of  operational  testing  is  to  provide 
information  for  use  in  an  independent  evaluation  of  the 
military  utility,  operational  effectiveness  and  suitability 
of  the  total  system  [1].  There  are  three  sequential  tests, 
OTI ,  OTII,  OTIII,  characterized  by  emphasis  on  testing  with 
typical  user  operators,  crews,  or  units  under  realistic 
conditions.  Each  sequential  test  consists  of  several  trials. 
Data  obtained  during  a  particular  segment  of  the  sequence  is 
analyzed  to  determine  if  the  next  phase  of  the  test  should 
be  conducted  or  the  new  system  rejected  [2]. 

Operational  test  I,  (OTI),  usually  is  limited  in 
scope  and  focuses  on  the  primary  system  function  (i.e., 
firepower  of  a  weapon,  mobility  of  a  transport  system,  etc.). 
The  type  of  comparison  is  either  against  a  baseline  system 
or  among  competing  systems.  Operational  test  II,  (OTII) 
is  broader  in  scope  and  is  concerned  with  testing  of  engi¬ 
neering  prototype  equipment  and  complete  test  support 
packages  involving  entire  troop  units  in  controlled  field 
exercises.  The  comparison  is  between  the  new  system  and  the 
standard  system  which  would  be  replaced.  Operational  test 
III  (OTIII),  involves  evaluating  the  performance  of  as  large 


a  unit  as  feasible,  employing  the  new  system  versus  the  same 
unit  employing  the  current  system  in  use.  It  is  important, 
therefore,  not  only  to  detect  if  learning  is  occurring  but 
to  detect  it  early  in  the  OT  before  additional  time  and 
money  is  expended  on  obtaining  possibly  meaningless  results. 

To  obtain  timely  information  for  deciding  to  stop  the  OT 
will  usually  require  an  on  site  evaluation  of  the  test 
results.  Thus  any  methodology  developed  must  not  only  be 
able  to  detect  learning  but  must  also  be  applicable  in  a 
field  environment. 

Fundamentals  of  the  Learning  Curve 

Assuming  the  performance  of  the  system  is  dependent 
on  operator  proficiency  and  can  be  described  by  a  monotonic 
function,  we  then  have  a  situation  which  can  be  modeled  by 
the  basic  learning  curve  function. 

Learning  defined  by  improved  cycle  time  or  performance 
over  repeated  trials  can  be  divided  into  two  distinct 
phases:  threshold  learning  and  conditioned  learning.  Thresh¬ 

old  learning  is  that  learning  which  occurs  prior  to  the  time 
the  operator  can  do  the  operation  from  memory.  Conditioned 
learning  is  that  learning  which  occurs  after  the  person 
remembers  how  to  perform  the  operation  without  relying  on  a 
trial  and  error  procedure.  For  this  research,  only  the  second 
phase  or  conditioned  learning  will  be  considered. 

According  to  the  findings  of  previous  research  studies 
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[3],  [5],  [8],  [9],  the  learning  process  can  be  defined 
by  an  equation  of  the  form: 


z  =  at 


(1-D 


where 


z  =  cycle  time 

a  =  a  constant  which  is  determined  by  the  cycle  time 
at  the  beginning  of  conditioned  learning 

t  =  the  cycle  number  from  the  start  of  conditioned 
learning 

b  =  a  constant  which  is  determined  by  the  rate  of 
learning  over  trials. 

Although  this  function  is  continuous  for  values  of  t  greater 
than  zero,  learning  can  only  be  meaningfully  evaluated  at 
discrete  values  of  t.  This  particular  equation  describes 
the  learning  of  an  operation  without  any  interruption  of 
significant  duration  which  could  have  a  negative  effect  on 
previously  learned  information  and  skills.  The  values  of  the 
parameters  will  always  be  greater  or  equal  to  zero. 

In  conducting  trials  during  a  particular  phase  of  the 
operational  test  at  OTEA,  the  time  between  trials  sometimes 
varies  but  it  is  believed  to  have  no  significant  effect  on 
retention  of  previously  gained  knowledge  and  skills. 

Recently  much  interest  has  been  focused  on  group 
learning  patterns.  Several  case  studies  have  been  conducted 
to  determine  if  group  learning  can  be  described  by  an 
equation  similar  in  form  to  equation  (1-1).  Although  studies 


in  this  area  have  been  somewhat  limited  in  their  scope,  it 
appears  on  the  surface  that  team  learning  exhibits  the  same 
performance  curve  displayed  by  individual  operators  [12]. 

Another  way  to  examine  learning  is  by  a  performance 

curve 

2  =  1 -  at  "b  (1-2) 

This  curve  is  based  on  the  same  theory  as  the  "cycle  time" 
curve  except  the  asymptote  of  this  function  approaches  the 
value  1  (see  Figures  1-1  and  1-2).  The  performance  curve 
is  based  on  percent  achieved  from  the  total  possible  obtain¬ 
able.  This  method  of  recording  learning  would  be  appropriate 
when  accuracy  rather  than  time  to  completion  was  the  primary 
objective.  The  restrictions  on  parameter  "b"  are  the  same 
as  for  the  learning  curve,  however,  parameter  "a"  will  only 
take  on  values  in  the  interval  [0,1]. 

Although  the  theoretical  asymptote  for  the  curve  is 
one  when  the  number  of  trials  approaches  infinity,  this 
function  can  approach  any  value  between  zero  and  one  as  a 
working  asymptote  by  using  the  proper  combination  of  parame¬ 
ters  "a"  and  "b."  A  working  asymptote  is  referred  to  here  as 
that  value  on  the  curve  where  the  change  in  performance 
between  trials  is  so  small  that  it  would  be  considered 
negligible  for  practical  purposes. 

For  example,  in  a  given  trial  of  an  OT  if  a  weapon 
is  fired  at  a  target  100  times,  the  total  possible  performance 
would  be  100  hits  or  100  percent.  If  the  weapon  is  only 
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capable  of  hitting  75  percent  of  the  targets  when  fired  by 
fully  learned  operators,  then  the  function  describing  the 
learning  of  the  operators  over  several  trials  would  approach 
an  asymptote  of  .75  on  the  performance  scale.  The  theoretical 
asymptote  of  the  function  describing  learning  would  of 
course  still  be  one;  however,  for  evaluation  purposes  the 
operators  would  be  considered  fully  learned  and  the  weapons 
capability  assumed  to  be  less  than  100%  accuracy. 

In  both  curves,  equation  (1-1)  and  equation  (1-2), 
the  value  of  the  function  before  the  first  trial  and  between 
discrete  values  of  the  trials  has  no  significance  since  there 
is  no  measure  of  knowledge  until  a  trial  is  completed. 
Therefore,  when  dealing  with  learning  curve  equations, 
whether  cycle  time  or  performance  oriented,  the  primary 
concern  is  the  description  of  learning  over  discrete  values 
of  t  >  1 . 

Basis  for  a  Performance  Curve  in  Test  Results 

Before  conducting  any  operational  tests  at  OTEA, 
the  participants  undergo  a  thorough  condensed  training 
program  on  the  system  to  be  evaluated.  Due  to  extremely 
high  costs  in  operating  the  system,  much  of  the  training  is 
conducted  piecemeal  under  simulated  conditions  which  may  or 
may  not  truly  represent  the  performance  of  a  crew  in  an 
actual  situation.  If  a  crew's  performance  could  be  improved 
by  actually  operating  the  system,  then  the  test  results  over 
several  trials  of  an  OT  segment  should  reflect  improvement 


through  better  scores.  If  a  sufficient  number  of  trials 
were  conducted  the  test  results  would  eventually  level  off 
indicating  the  crew's  performance  has  peaked  and  that  no 
further  learning  is  taking  place.  Thus  it  seems  reasonable, 
and  OTEA  test  results  are  currently  being  evaluated  in 
support  of  this  conclusion,  that  a  performance  curve  function 
can  be  fit  to  test  results  of  this  nature. 

Objective 

The  objectives  of  this  research  are  two  fold.  The 
first  objective  is  to  devise  a  field  expedient  methodology 
for  testing  if  the  rate  of  learning  is  significantly  different 
from  zero.  The  second  objective  is  to  develop  a  methodology 
which  can  measure  the  rate  of  learning  taking  place  at  any 
particular  trial. 

General  Approach 

Since  it  is  well  documented  that  the  curve  describing 
performance  as  learning  progresses  follows  an  asymptotic 
curve,  the  rate  of  learning  then  could  be  analyzed  by  either 
evaluating  the  first  derivative  of  a  curve  fit  through  the 
data  points  or  examining  the  slope  of  a  line  between  data 
points  for  specified  times.  If  the  first  derivative  of  the 
curve  or  the  slope  of  the  line  is  positive,  it  is  an  indication 
that  learning  is  taking  place.  One  procedure  to  be  examined 
will  be  to  estimate  the  slope  of  a  linear  model  fit  to  the 
observations  and  test  for  its  statistical  significance. 
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The  test  procedure  involves  simple  algebra  and  requires  no 
information  about  the  parameters  of  the  actual  curve. 

The  test  results  on  the  slope  using  a  linear  model 
would  only  indicate  if  learning  occurred  during  the  trials 
and  can  provide  no  information  as  to  whether  the  rate  of 
learning  was  decreasing  over  the  latter  trials. 

Another  drawback  using  linear  methods  is  that  the 
rate  of  learning  will  be  tested  using  an  estimate  of  the 
variance  obtained  by  fitting  a  linear  model  of  the  form 

Yi  =  c  +  dt i  +  «i  (1-3) 

where 

yi  represents  the  obseivation  at  trial  i 
t^  represents  trial  i 
c  is  the  intercept  value 
d  is  the  slope  of  the  line 

6.  difference  between  the  observation  and  the  line 
1  at  trial  i 

when  the  true  model  is  the  nonlinear  function 

y^  =  1-at  b  +  (1-4) 

The  error,  ,  between  any  observation,  y^,  and  its 
expected  value,  z^,  is  assumed  to  be  independent  and 
normally  distributed  with  expected  value  of  zero  and  variance 
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2  2 

of  a  ,  NID  (0,  a  ).  If  the  error  term  such  as  e.  is  a  sum 

e  e  l 

of  errors  from  several  sources,  then  no  matter  what  the 
probability  distribution  of  the  separate  errors  may  be, 
their  sum  e^  will  have  a  distribution  that  will  tend  more 
and  more  to  the  normal  distribution  as  the  number  of 
components  increases  by  the  central  limit  theorem  [13]. 

An  error  in  a  test  observation  may  be  a  composite  of  a 
scoring  device  error,  an  error  due  to  a  small  leak  in  the 
system,  an  error  due  to  an  unexpected  physical  ailment 
affecting  the  operator,  an  error  due  to  changes  in  wind 
velocity  and  so  on.  The  components  of  this  error  term 
would  not  include  those  dependent  on  operator  proficiency 
and  likely  to  decrease  with  additional  repetitions  or  training. 
This  latter  type  of  error  is  often  used  to  record  learning 
and  would  be  reflected  by  the  performance  curve.  The  error 
terms  in  equation  (1-3)  may  be  larger  than  in  equation  (1-4) 
due  to  a  lack  of  fit  of  the  model  which  will  in  turn  inflate 
any  estimate  of  the  variance  used  in  testing  for  the 
statistical  significance  of  the  slope,  d.  The  closer  the 
trial  observations  are  to  the  asymptote  of  the  expected 
curve,  however,  the  better  the  estimate  of  the  variance  will 
be  since  the  lack  of  fit  component  will  be  decreasing. 
Therefore,  a  linear  method  may  be  appropriate  to  detect 
learning  if  the  estimate  of  the  variance  is  relatively 
accurate . 

Another  approach  will  be  to  fit  a  curve  to  the 


observations  using  a  nonlinear  regression  technique  and 
analyze  the  location  of  the  trial  results  in  relation  to  the 
fitted  curve.  Although  the  estimate  of  the  variance  using 
nonlinear  techniques  will  be  more  accurate  than  that  using 
the  linear  estimate,  the  difficulty  in  conducting  signifi¬ 
cance  testing  is  that  the  estimates  obtained  using  nonlinear 
techniques  do  not  have  the  linear  properties  necessary  to 
conduct  the  known  significance  tests.  It  may  be  possible 
however  to  use  linear  theory  results  as  approximations  for 
determining  a  confidence  region  for  the  parameters  of  the 
nonlinear  model  if  the  degree  of  nonlinearity  is  not  too 
large.  If  the  performance  function  satisfies  this  require¬ 
ment  then  the  rate  of  learning  can  be  determined  by  analyzing 
the  approximated  confidence  interval  about  the  slope  at 
specified  trials.  This  procedure  then  would  provide  a  means 
to  determine  how  close  to  being  fully  learned  the  operators 
are  at  each  trial. 

Measure  of  Nonlinearity 

When  a  model  is  nonlinear  there  is  an  estimation 
space,  however,  it  is  not  defined  by  a  set  of  vectors  and 
may  be  quite  complex.  If  the  estimation  space  consists  of 
all  points  with  coordinates  (f(x^,0),  fCx^.Q),  .  .., 
f(*m,9)}  then  minimizing  the  sum  of  squares  function 
ss(9)  corresponds  geometrically  to  finding  a  point  p  on  the 
estimation  space  which  is  the  shortest  distance  to  Y,  the 
vector  of  observations. 
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A  sample  space  for  a  very  simple  non-linear  example 

involving  only  n=2  observations  and  y2  taken  at  x  =  Xj 

and  x  =  *2>  respectively,  and  a  single  parameter  0  is 

illustrated  by  Draper  and  Smith  [6]  and  is  reproduced  in 

Figure  1-3.  The  non-linear  estimation  space  consists  of  the 

curved  line  which  contains  points  {ffx^e),  f(x2,0)  as  the 

parameter,  e,  varies,  and  the  independent  variables  x^,  x2 

are  fixed.  The  point  Y  has  coordinates  (y1,y2)  and  p  is 

the  point  of  the  estimation  space  closest  to  Y.  When  the 

linearization  technique  is  applied  to  a  non-linear  problem, 

a  new  origin  is  selected,  say  6  ,  and  a  linearized  estimation 

space  in  the  form  of  the  tangent  line  at  0q  is  then  defined. 

The  linear  estimation  space  contains  the  points  (f(x, ,0  )  + 
3£(x1P0)  9£(x2,0)  1  ° 

- go -  ,  f(x2,0Q)  +  - gg -  as  0  varies  and  x^ ,  x2  are 

fixed.  However  if  the  rate  of  change  of  f(x,0)  is  small  at 

0Q  but  increases  rapidly,  the  units  on  the  tangent  line  may 

be  unrealistic  in  terms  of  determining  good  estimates  of  the 

parameters  that  will  minimize  the  sum  of  squared  errors 

between  the  observations  and  the  proposed  model.  Again 

Draper  and  Smith  give  an  excellent  illustration  of  gross 

inequalities  in  the  systems  of  units.  See  Figure  1-4. 

In  Figure  1-4  the  best  linear  approximation  of  the  true 
parameter  solution  from  the  point  0  =  0Q  is  the  point  0  =  Q0 . 
It  is  obvious  that  if  the  linear  solution  0  =  Q0  is  used 
as  the  next  starting  point  on  the  estimation  space  we  will 
be  further  from  the  best  point  P  then  was  our  original 
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guess  6  =  eo  =  0.  If  the  degree  of  non-linearity  is  not  too 
large,  it  may  be  possible  to  use  linear  theory  results  to 
approximate  the  confidence  region  for  the  nonlinear 
function.  Therefore  we  need  a  procedure  that  will  determine 
when  linear  theory  results  provide  acceptable  approximations 
to  the  nonlinear  estimation  problem. 


igure  1-3.  Geometric  Interpretation  of  Linearization 
Method  (n=2,  0=1) 


Lstimation  Space  / 


Figure  1  -  1  . 


Lffect  on  the  Linearization  Method  of  Gross 
Inequalities  in  the  System  of  Units 
fn=2 ,  0=1) 
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CHAPTER  II 

METHODOLOGY 

The  curve  describing  operator  performance  on  a 
particular  system  is  well  defined  in  terms  of  the  parameters 
"a"  and  "b."  This  curve  can  then  be  used  as  a  basis  for 
determining  any  given  crew's  performance  level  on  the  system. 

Determining  the  Distribution  of  the  Linear  Test  Statistic 

A  method  for  determining  a  crew's  performance  level 
when  the  curve  is  well  defined  is  to  examine  the  expected 
values  of  the  observations.  The  procedure  would  be  to  find 
the  expected  value  for  each  trial  observation  by  minimizing 
the  sum  of  the  squared  errors  between  the  observations  and 
the  known  curve  at  discrete  consecutive  trial  numbers.  If 
the  expected  value  that  corresponds  to  the  last  observation 
is  at  the  asymptote  of  the  curve  it  is  assumed  the  crew  is 
fully  learned.  This  procedure  is  summarized  as  follows. 

Given  n  observations  denoted  as  y^,  y  2,  •  ••>  yn>  label 
their  respective  trial  numbers  k,  k+1,  ...,  k^ and  find 
the  discrete  value  of  k  that  will  minimize  the  sum  of 
squared  errors  between  the  observations  and  values  on  the 
curve  computed  at  the  corresponding  observation  trial  numbers. 

The  procedure  for  this  is  to  find  the  discrete  value 


of  k,  kQ,  on  the  known  curve  that  corresponds  to  the  minimum 
error  between  the  first  observation  and  the  value  of  the 
known  function  (see  Figure  2-1).  Retaining  the  same  labeling 
for  the  trial  numbers  corresponding  to  the  observations, 
conduct  a  search  over  discrete  values  in  the  vicinity  of 
kQ,  and  find  the  discrete  value  for  k  that  minimizes  the  sum 
of  squared  errors  (Figure  2-2).  Find  the  corresponding  trial 
number,  k  +  N  -  1,  for  the  last  observation,  and  then  compute 
value  of  the  known  function  at  this  value.  If  the  value  of 
the  function  at  k  +  N  -  1  is  the  asymptotic  value,  this 
corresponds  to  the  situation  in  which  the  expected  value  of 
the  last  observation  is  the  asymptote  which  means  a  fully 
learned  status. 

Another  approach  to  determining  the  performance  level 
would  be  to  conduct  statistical  tests  on  the  observations. 

If  the  distribution  of  the  error  between  an  observation,  y^, 
and  its  expected  value,  zi ,  is  MID  (0,  a£2),  then  the 

distribution  of  the  observations  at  a  given  trial  is  MID 

2 

^zi’  ae  ^ue  ^he  nOFmality  assumption. 


yi  -  Zi 

VT “ 

£ 

where  a  is  an  estimate  for  the  true  variance 


(2-1) 


follows  a  student-t  distribution.  To  test  for  learning  over 
a  series  of  trials,  the  test  statistic  would  be 
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where 

7.  =  asymptotic  value 

y  =  average  value  of  the  observations 
since  the  expected  value  for  y  is  z  ,  when  the  observations 
are  at  the  asymptote.  If  the  variance  is  unknown,  then  an 
estimate  must  be  found  which  will  provide  an  unbiased 
estimate  of  the  variance  in  order  to  conduct  the  test 
described  in  equation  (2-2). 

This  is  the  basis  for  conducting  the  linear  tests 
discussed  in  Chaoter  I.  If  E(y)  =  z  ,  then  all  the  observa¬ 
tions  are  at  the  asymptote  and  the  rate  of  change  between 
observations  or  the  slope  of  the  linear  model  will  be  equal 
to  zero.  The  discussion  that  follows  in  the  remainder  of  the 

section  and  in  the  next  section  will  be  devoted  to  obtaining 

2 

an  estimate  of  o£  in  order  that  the  t-test  may  be  used  to 
test  for  learning. 

Assuming  then  that  the  error  term,  in  any 

observation,  y^ ,  which  fits  the  form 

y.  =  l-at"b+e.  (2-3) 

is  NID(0,  cr  ^),  a  minimum  variance  unbiased  estimate  for  the 
variance  of  an  observation  about  its  expected  value  would  be 
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£  (y^O 

Sy2  =  -  (  2-4  ) 

y  N 

where  is  the  i^^  trial  observation 

is  the  expected  value  of 

Since  in  most  cases  the  expected  curve  of  the  obser 
vations  will  be  unknown,  this  procedure  is  of  little 
practical  value. 


Estimating  a  When  the  Paramters  Are  Unknown 


Continuing  with  the  performance  curve  as  defined  in 

* 

equation  (1-2),  performance  will  reach  an  asymptote  as 
the  number  of  trials  increase.  There  is  then  some  critical 
trial  number,  tc,  where  further  trials  will  show  negligible 
improvement  in  proficiency. 

Consider  a  situation  where  no  well  defined  curve 
exists.  The  expected  values  of  the  observations  are 
unknown.  To  estimate  the  variance  using  an  equation  similar 
in  form  to  equation  (2-4),  an  estimate  for  z ^  must  be 
obtained.  Consider  the  curve  in  Figure  2-3. 


In  the  remainder  of  this  research,  the  word  asymptote 
refers  to  the  value  of  the  curve  where  the  rate  of  change 
over  future  trials  is  so  small  (say  .00001)  that  it  is 
considered  zero  for  practical  purposes. 


curve  beyond  trial  tc,  the  differences  in  their  expected 
values  would  be  small  enough  to  be  considered  zero  for  all 


practical  purposes.  The  expected  value  for  each  observation 


is  then  considered  to  be  equal  to  zQ,  the  value  at  the 
asymptote  of  the  curve.  Since  the  observations  in  this 
situation  are  all  distributed  NID  (zc,  ),  an  unbiased 
estimate  for  z^  in  equation  (2-4)  would  be  the  average  of 
the  observations. 


N 

7  =  2  y./N 

i=l  1 

where  +  ei 


(2-5) 


E(y)  =  E[  Z  (z- +e. )/N] 
i  =  l 


E(y)  = 


I  E(z.+e.) 
i  =  l  1 


E(y) 


Z  z- 
i  =  l  1 


but  z . 
1 


therefore 


ECy)  =  N  zc/N  =  zc 


Thus  an  efficient  unbiased  estimate  for  the  variance  [ll] , 


using  the  sample  average  as  the  estimate  of  the  z^'s 
the  observations  are  all  at  the  asymptote  is 


when 


2  i  =  l 


e  (y--y) 


When  the  observations  are  not  at  the  asymptote,  this  esti¬ 
mate  of  the  variance  will  be  inflated  since  the  average  of 
the  observations,  y,  will  no  longer  be  an  unbiased  estimate 
for  the  expected  value  of  each  observation.  The  amount  of 
bias  will  be  a  function  of  the  distance  that  the  expected 
values  of  the  observations  are  from  the  asymptote  of  the 
true  curve. 

Since  the  performance  curve  function  is  non-linear, 
there  is  no  easy-to-apply  procedure  to  obtain  an 
unbiased  estimate  for  the  variance  when  the  parameter 
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values  are  unknown. 

Several  alternative  techniques  for  estimating  the 

variance  will  be  examined  in  terms  of  the  expected  bias 

to  determine  a  minimum  biased  estimator.  The  deviation  of 

several  estimators  is  contained  in  Appendix  B. 

The  three  estimators  to  be  investigated  for 

estimating  a  2  are: 

e 


(OBsr  =  z  (y,-y)  /N-i 

i=l 


(2-7) 


(SEX)2  =  (N-l )  Z  ( x . -x ) 2/2N (N-2 ) 
i=l 


(2-8) 


(SER)2  =  (N2+l) (N+2) (MSe)/(N3-2N2+N+1 )  (2-9) 


In  summary,  the  expected  values  for  the  respective  variance 
estimators  are: 
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Note  that  if  the  expected  values  of  the  observations,  the 
z^'s,  were  all  at  the  asymptote,  the  bias  factor  in  all 
three  equations  is  zero.  As  the  difference  between  the 
expected  values  of  the  observations  and  the  asymptotic  value 
of  the  curve  increases  however,  the  expected  bias  value 
associated  with  each  procedure  changes.  The  relative 
usefulness  of  the  estimators  will  be  numerically  analyzed 
in  Chapter  III. 


2 

Variance  of  the  Estimators  of  aE 

2 

If  the  variances  of  the  estimators  of  a£  are  not 
significantly  different,  then  the  expected  bias  may  be  the 
only  criteria  necessary  to  determine  the  best  estimator 
(see  Figure  2-4a) .  If  on  the  other  hand  the  variances  of 
the  estimators  differ  significantly,  then  it  is  possible  that 
the  minimum  biased  estimator  is  not  the  best  estimator  in 
terms  of  the  percent  of  estimates  within  the  specified 
tolerance  limits  (see  Figure  2-4b) . 

When  the  allowable  error  tolerance  for  the  variance 
estimate  is  5,  then  estimator  1,  in  Figure  (2-4b)  ,  is  the 
better  estimator.  When  the  allowable  error  is  26,  however, 
then  estimator  2  is  better  since  it  has  the  largest  percent 
of  its  estimates  within  the  tolerance  limits. 

The  expected  bias  associated  with  each  estimator  is 


a  function  of  the  expected  values  of  the  observations  and  is 


Figure  2-4a.  Possible  Distribution  of  Estimates  when  the 
Variances  of  the  Estimates  are  Equal 


Figure  2-4b.  Possible  Distribution  of  Estimates  when  the 
Variance  of  the  Estimators  are  not  Equal 
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only  affected  by  changes  in  the  parameter  values  or  the  sample 

size.  Therefore,  the  difference  between  the  expected  value 

2  2 

of  the  estimate  for  a  and  the  value  of  a  for  specified 

e  e 

2 

a,  b,  and  N,  will  not  change  as  the  size  of  changes. 

2 

The  magnitude  of  a£  will  be  a  factor  however,  in 

determining  the  expected  bias  in  the  estimate  of  o  .  See 

Figures  ( 2-5  )  and  (  2-6  )  .  This  is  important  since  this  is 

the  value  that  will  be  used  to  compute  any  statistical  test 

on  the  slope.  As  the  process  variance  increases  for  specified 

parameter  values  and  sample  size,  the  expected  value  of  the 

2 

estimate  of  a£  approaches  the  true  value  of  .  As  o£ 

increases  though,  there  will  be  a  corresponding  increase  in 

the  variance  of  the  estimator.  This  will  affect  the 

dispersion  of  the  estimates  and  thus  could  also  affect  the 

solution  of  an  estimator.  See  Figure  (2-5)  and  Figure  (2-6). 

2 

If  the  contribution  of  a  to  the  estimator  variance 

e 

for  one  estimator  is  larger  than  for  another,  a  situation 

could  also  occur  as  depicted  in  Figure  (2-7a)  and  Figure  (2-7b) 
2 

When  o£  is  small  as  in  Figure  (2-7a),  estimator  2  is 

2 

better,  however,  in  Figure (2-7b)  where  a£  is  large,  estimator 
1  appears  to  do  as  well  or  better  than  estimator  2.  To 
determine  a  best  estimator  then,  the  effect  of  the  variance 
on  the  estimates  as  well  as  that  of  the  parameter  values  and 
sample  size  must  be  examined. 


Estimator  2 


Figure  2-5.  Expected  Bias  Large  and  Dispersion  of 

Estimates  Small  when  a  is  Small 

e 


Estimator  2 


Figure  2-6.  Expected  Bias  Small  and  Dispersion  of 
Estimates  Large  When  is  Large 


An  evaluation  of  the  variance  of  the  estimator  as 
well  as  the  expected  bias  factor  will  thus  be  required  before 
selecting  the  best  estimator  of  a£  .  An  effort  was  made  to 
obtain  a  closed  form  expression  for  the  variance  of  each 
estimator  in  terms  of  the  true  variance,  ,  and  the  expected 
values  of  the  observations,  the  z^'s.  Due  to  the  complex 
forms  involved,  this  approach  was  abandoned  in  favor  of 
analysis  by  computer  simulation.  The  results  of  the  simula¬ 
tion  study  will  be  presented  in  Chapter  III. 


Small  a£ 

Figure  2-la..  Possible  Effect  on  Dispersion  of  the  Estimates 

of  a  as  the  True  Value  of  a  Increases 
e  t 


Linear  Methods  to  Test  for  Learning 


Since  the  performance  curve  describing  the  progress 
of  training  is  an  asymptotic  function,  it  may  be  possible 
to  determine  when  a  unit  is  fully  learned  by  examining  the 
slope  of  a  line  fit  through  the  trial  observations  or  examin- 
the  rate  of  change  between  the  observations.  If  the  rate  of 

change  is  large,  then  learning  is  taking  place  and  when  the 
rate  of  change  approaches  zero,  a  very  small  amount  of  learning 
is  occurring  which  corresponds  to  approaching  a  "fully 
learned"  status.  An  appropriate  method  to  determine  the 
level  of  performance  then  would  be  to  test  if  the  slope  of 
the  linear  model  fit  through  the  observations  or  the  rate 
of  change  between  observations  is  significantly  greater  than 
zero . 

Linear  Methods  of  Analysis 

As  discussed  earlier,  if  the  rate  of  learning  is  large 
and  the  error  variance  of  reasonable  size,  it  may  be  possible 
to  detect  learning  by  examining  the  slope  using  linear 
approximation  methods.  Several  methods,  which  can  be 
solved  by  simple  hand  calculations,  are  examined  using  the 
variance  of  the  slope  as  a  basis  for  comparison  in  selecting 
which  is  best.  Those  linear  approximation  methods  which 
provide  the  smallest  variance  for  the  slope  will  be  selected 
for  further  study  and  possible  application. 
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a.  "Average  of  Two  Groups  (ATG)".  In  order  to 
obtain  an  estimate  of  the  slope  it  is  necessary  to  have  at 
least  two  points.  The  average  of  two  groups  method  divides 
the  data  at  the  midpoint  of  the  trials  into  two  groups.  The 
average  of  the  observations  is  computed  for  each  group,  and 
the  difference  between  the  averages  tested  to  determine  if 
this  difference  is  significantly  greater  than  zero.  If  there 
were  an  odd  number  of  observations,  the  middle  observation 
would  not  be  considered  in  the  computations. 


Let  DE  =  average  of  observations  in  first  group 
=  average  of  observations  in  second  group 

dATG  DL  ~  °E  (2-10) 


then 


Var(dAT(J)  =  Var(DL-DE) 

N/2  N 

Var  (d  Arr,p  ^  =  Var  (  Z  y,/(N/2))  +  Var(  £  y  ./(N/2)) 

ArG;  i=l  1  i-N/2+1  x 

(M/2)o 2  (N/2)a  2 

Var  (dflrrp  )  =  - §-  +  — - — - 

hilt  / .  T  /«  \  <:  /TkT/^\^- 


(N/2)1 


(N/2)' 


4a 


Var(dATG) 


(2-11) 


N 


b.  ''Average  of  Consecutive  Differences  (ACD)".  In 
this  method  the  average  of  the  differences  between  consecutive 
observations  is  analyzed  to  determine  if  learning  is 
occurring . 


Then  computing  the  difference  between  the  average  obtained 
in  the  latter  group  with  that  of  the  earlier  group  one 
obtains  an  estimate  of  the  slope  with  corresponding  variance 
as  follows: 


Z  (yi+l"yl) 
1=1  11  1 


2  (yi+l_yi) 
N/(2+l)  1 


dATGCD  ~  °L"  °E 


(2-1*0 


Var(dATGCD)  Var(DL  de)  ~  (n_2)2  (2_15) 
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As  in  the  ATG  method,  if  there  is  an  odd  number  of  observations, 
the  middle  observation  would  be  discarded. 

d.  "Linear  Least  Squares  Regression  (LLSR)".  This 
method  fits  a  line  through  the  observations  such  that  the 
total  of  the  distances  squared  of  the  observations  from  the 
line  is  minimized.  Draper  and  Smith  [6]  provide  an  excellent 
description  of  the  details  involved.  A  brief  summary  of  the 
procedure  is  given  later  in  this  Chapter. 

An  estimate  of  the  slope  is  defined  as: 


N 

E  (t,-t)(z  ) 
-  i=l 

LLSR  N  5 

X  (t,-tr 

i=l 


(2-16) 


where  ti  =  trial  number  i 

z.  =  expected  value  of  the  observation 
at  time  i 


and  the  slope  variance  is  defined  as 


Var(dLLSR)  N  ?  N  ?  _p 

E(t,-t )  Zt  *  -  2t  £t . +Nt ^ 

i=l  1  i=l 1  1 


(2-17) 


Since  the  trials  are  consecutive  from  1  to  N  the  following 
closed  form  expressions  for  t  can  be  used  in  the  above  equation 


!  4  2  N(N+1 ) ( 2N+1 ) 

w'i  =  6 


t 


N+l 

2 


N (N+l ) 
2 


Then 


Var^dLLSR^ 


Var(dLLSR) 


■)+N(^|i)2 


N  (N  +1 ) 


(2-19) 


For  N>2,  the  linear  last  squares  regression  method  provides 
the  best  estimates  for  the  variance  of  the  slope.  See  Figure 
(2-8).  The  average  of  consecutive  differences,  ACD,  method 
was  the  next  best  procedure.  The  average  of  two  groups 
method  becomes  the  third  best  procedure  when  N=8,  but  the 
estimate  of  the  variance  of  the  sloDe  is  still  quite  large  in 
comparison  to  the  ACD  and  the  LLSR  methods. 

From  the  several  methods  considered  for  detecting 
learning  through  analyzation  of  the  slope,  the  two  best 
procedures,  using  the  minimum  variance  as  the  selection  cri¬ 
teria,  appears  to  be  the  average  of  consecutive  differences 
(ACD)  method  and  the  linear  least  squares  regression  (LLSR) 


method . 


To  complete  the  analysis  of  the  linear  approximation 


procedures,  the  expected  value  of  the  estimate  of  the  slope 
using  the  ACD  method  and  the  LLSR  method  will  be  examined. 
Let  l  represent  the  true  average  rate  of  learning  for  the 
process  over  N  trials.  An  expression  for  2.  then  is 


i  mppp  j  i  iu  m 


1.5a, 


ATG  Method 


ACD  Method 


LLSR  Method 


Hfrimjiiiiri  m  *jk+a***. 


ATGCD  Method 


2  3  5  5  6 

Number  of  trials 

Figure  (2-8).  Variance  of  the  Average 
Slope  Estimates 

,  -  ZN'Z1 

*  ~ TT 1 


a-at. 


(2-20) 


The  expected  value  of  an  estimate  of  Z  using  the  ACD 


method  is 


Etd.r-J  ' 


»  yN-yi 

Et<W  ' 


crCzH^N)-(Zi  +  ei)i 

fc  l - nrm - J 


C1'atN  b+EN)  '  Cl-at1'b+e1) 


a-at. 


E(<W 


since  =  1  and  E(e^)  =  0 


(2-21) 


As  expected,  the  estimate  of  the  slope  using  the  ACD  method 
is  unbiased. 

The  expected  value  of  an  estimate  of  2.  using  the  LLSR 
method  is  derived  as  follows: 


E(dLLSR]  *  E 


Z  (ti-t)yi 


l  (t.-t)' 


Z  (t-t) 


EtCtj-t) (z1+e1)+(t2-t) (z2+e2)  + 


+  (zN+eN)J 


^  -  2 
i=l  1 


[  (ti-t)  Z1+(t2-t)  (Z2  +  ...  +  ( t^  -  t )  Zjj] 


Replacing  the  Et . ’ s  with  expressions  in  terms  of  N  we  get 
i=l 


1 


36 


12 

N(N2+1) 


I  Ct1 


+  (t 


N+l 


N 


)*N1 


12 

71 


N  (N  +1)  i=l 


Z  tizi  -  Z  z • 


- 2 — 

N (N  +1)  i  =  l  1 


Substituting  1-at^ 


for  z^  we  get  an  expression  of  the  form: 


12 

N(N2  +  1) 


Z  t.  (1 -at . 'D) 
i=l  1  1 


6(N+1) 
N(N2  +  1) 


Z  (1- at . ~ D) 
i  =  l  1 


12 

N(N2+1) 


N 
[  S 
i  =  l 


t. 


N 

-a  Z  t. 
i=l  1 


-b+1  ^ 


.  6CN+1)  a  N  t  -b 
N(N2+1)  N(NZ  +  1)  i  =  l  1 


E ^dLLSR^ 


12a 


N(N4  +  1) 


N 

(N+l)  I  t. 
i=l  1 

- 2 - 


N 

Z 

i  =  l 


ti'b+1j 


(2-23) 


Subtracting  the  true  value  of  the  average  rate  of  learning 
from  the  estimate  in  equation  (2-23)  we  can  obtain  the 
expected  bias. 

Let  g  represent  the  amount  of  bias  in  equation  (2-23), 

then 


8  = 


12a 

7T 


(N+l) 


N 

iVi 


-b 


N 


N  (N  +1) 


T 


i=l 


t.--b+1) 


-  r- 


a-at 


N 

TTT 


(2-24) 


When  the  value  of  b  equals  zero  In  equation  (2-24)  which 
corresponds  to  the  expected  values  of  the  observations  all 
being  equal  to  the  asymptotic  value,  the  expected  bias  is 
zero.  The  bias  factor  in  equation  (2-24)  is  negative  which 
means  we  should  expect  the  estimate  for  the  average  rate  of 
learning  using  the  LLS  method  to  be  less  than  the  value 
defined  in  equation  (2-21). 

In  summary,  we  now  have  two  methods  to  test  for 
learning  which  exhibit  the  following  characteristics: 

(i)  The  LLSR  method  provides  a  biased  estimate  for 
the  average  rate  of  learning;  however,  it  has 
the  smallest  slope  variance  of  all  methods  examined, 
(ii)  The  ACD  method  provides  an  unbiased  estimate  for 
the  average  rate  of  learning  but,  the  slope 
variance  is  larger  than  that  in  the  LLSR  procedure. 
The  bias  factor  in  equation  (2-24)  will  be  computed 
and  its  effect  analyzed  for  various  combinations  of  parameter 
values  and  sample  size  in  Chapter  III.  A  discussion  of  the 
two  best  linear  test  procedures  is  in  the  next  two  sections. 

Average  of  Consecutive  Differences  (ACD)  .Method 

If  the  test  observations  are  at  the  asymptote  of  the 
curve,  the  difference  between  the  observations  will  follow 

2 

a  normal  distribution  with  mean  of  zero  and  variance  of  2a£  . 
Given  N  trial  observations  with  no  information  concerning  the 


true  variance  or  the  parameters  of  the  actual  curve,  we 
examine  the  average  of  the  consecutive  differences  in  the 
observations  which  corresponds  to  the  average  rate  of 
learning  over  the  N  trials. 

Define  x^  as  the  difference  between  observation 
yi+1  and  y. .  Then 


N-l 

E(x)  =  E  [  E  x. /N-l] 
*  _  1  1 


E(x)  =  El  E  Cyi+-.-yi)/N-i] 
i=l  11  1 


■  rr 


E(x)  fpx  iE1  (Ic'zc^ 


zc  =  expected  value  of  an 


observation  at  the  asymptote 


E (x)  =  ux  =  0 


From  equation  (2-13)  the  variance  of  the  average  of  the 
consecutive  differences  was  defined  as: 


Var  x  =  2ae^/(N-l)2 


where  x  =  d. _ 


Using  an  unbiased  estimate  of  o£  ,  the  test  statistic  for  the 
differences  will  follow  the  t -distribution .  A  test  to 
determine  if  the  average  slope  between  observations  is 
significantly  different  from  zero  would  be: 

Hq  :  slope  <  0  fully  learned 


H.:  slope  >  0  not  fully  learned 


Compute : 


*0  = 


2Sy  /  (N-l) 


(2-25) 


Testing  at  a  significance  level  of  a. 


If  t  >  t  ,,,  , 
o  a  N-l 


Reject  H 


If  t  5  t  ,.r  , 
o  a’N-l 


Do  not  reject  H 


Since  it  is  quite  possible  that  the  trial  observations 
will  not  all  be  at  the  asymptote,  our  estimate  of  the 
variance  will  be  inflated  due  to  bias  as  discussed  earlier. 
Using  a  biased  estimate  *of  the  variance  will  reduce  the  power 
of  the  test  which  is  defined  as: 


Power  of  test  =  1.0-P  [failing  to  reject  H0  when  H  is  false] 
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The  rationale  for  this  interpretation  is  that  using  an 
inflated  estimate  for  the  slope  variance  in  equation  (2-25) 
increases  the  probability  of  failing  to  reject  the  null 
hypothesis,  H  ,  when  it  is  not  true.  This  is  because  if  the 
test  statistic 


V2Est(a£2}/(N-l) 

is  greater  than  ta»N  2  usi-nS  3  biased  estimate  for  the 
variance  in  the  denominator,  it  will  remain  greater  than 
ta’N-2  w^en  an  unbiased  estimate  is  used.  On  the  other  hand, 
if  tQ  is  greater  than  ta>N_2  using  an  unbiased  estimate  for 
the  variance,  it  may  not  be  greater  if  a  biased  estimate  is 
used,  thus  creating  a  situation  where  we  would  fail  to  reject 
Hq  when  in  fact  it  is  false.  Therefore  if  the  test  statistic 
indicates  the  slope  is  significantly  greater  than  zero,  we 
can  assume  learning  is  occurring,  otherwise  the  results 
may  be  unreliable. 

A  summary  of  the  computations  required  to  conduct 
the  test  on  the  slope  using  the  ACD  method  follows: 

Compute  an  estimate  of  the  slope: 

a  1 

dACD  =  N-l  (2-26) 


Compute  an  estimate  for  the  standard  error  of  the  slope: 


mTmniiMl 


s.e.  (dACD) 


Compute  the  test  statistic: 


2[Est(cr 


(N-l ) ‘ 


t0  = 


dACD  0 

(dACD) 


Compare  test  statistic  at  desired  significance  level 


If  Pq  >  t^  Learning  is  occurring,  reject 


If  t„  5  t  „  „  No  conclusion 
0  a,N-2 


The  Linear  Least  Squares  Regression  (LLSR)  Method 


As  discussed  earlier,  this  method  involves  fitting 
a  linear  model  of  the  form 


yi  c  +  dLLSRti  +  6i 


(2-29) 


where  y^  =  value  of  the  observation 

t^  =  trial  number  that  corresponds  to 
c  =  y  intercept  value 

dT T oR  =  difference  between  the  observation  and 
the  line  at  trial  i. 

6.  =  difference  between  the  observation  and 
the  line  at  trial  i. 

through  the  observations,  based  on  minimizing  the  sura  of 
squares  of  deviations  of  the  observations  from  this  line. 

The  slope  of  the  line,  dTT<,,,,  at  the  asymptote  of  the 


curve  is  essentially  the  average  of  the  differences  between 
the  expected  values  of  successive  observations.  Therefore 
the  test  statistic  for  the  slope  of  the  line  using  the  LLSR 
method  follows  the  same  distribution  as  the  test  statistic 
for  the  differences  in  the  ACD  method,  namely  the  t-distri- 
bution  the  test  for  learning  would  then  be: 


Ho :  slope  <  0  Fully  learned 
:  slope  >  0  Not  fully  learned 


Compute 


*0  = 


dLL SR~° 

hi Lfi2-. 

N(N2  +  1) 


C  2  -  3  0 ) 


/v 

where  dLLSR  is  the  estimated  slope  for  the  linear  model. 


Compare  at  a  significance  level 

If  tQ  >  ta>N_2  Learning  is  occurring.  Reject 

If  t  <  t  „  No  conclusion 

o  a  N-2 

As  the  bias  factor  increases,  the  width  of  the  confidence 

interval  for  the  slope  increases  or  expressed  another  way, 

the  test  statistic,  t  ,  decreases. 

o 

As  was  previously  shown,  using  a  biased  estimate  for 


the  variance  versus  an  unbiased  estimate  when  testing  the 
slope,  equation  (2-28),  results  in  a  less  powerful  test. 


This  also  holds  true  for  the  estimate  of  the  average  rate 
of  learning.  As  was  the  case  with  the  ACD  method,  the  test 
results  may  be  unreliable  if  the  slope  test  statistic  is 


less  than  the  comparison  value,  t^,^  2‘ 

In  the  discussion  of  the  method  that  follows,  it  is 
assumed  that  the  reader  is  familiar  with  the  linear  least 
squares  regression  procedure.  The  LSSR  equations  required 
to  conduct  the  slope  test  are  presented  without  the  derivation. 
(For  a  development  of  the  equations ,  see  Draper  and  Smith 
[6].) 


The  slope  of  the  line  is  found  by  minimizing  the  sum  of 
the  squared  errors  between  the  observations  and  the  fitted 
line  equation  (2-29)  .  By  taking  the  first  derivative  of  the 
equation  for  the  sum  of  squared  errors  with  respect  to  each 
of  the  parameters  c  and  it  is  possible  to  solve  the  two 

resulting  equations  simultaneously  to  obtain  an  estimate 
for  the  slope. 


N 

T  (t.-t)y. 
^  _  i=l  1  1 

dLLSR  7" 

z  (t.-t)z 
i  =  l  1 


where 

t^  represents  trial  number  i 

y^  represents  the  observation  that  corresponds  to 
trial  i 


d 


in  equation 


t  is  the  average  of  the  t^ 

y  is  the  average  of  the  y ^ 

with  this  estimate  of  the  slope  and  replacing  S 

2 

(2-30)  with  an  estimate  for  a  we  can  conduct 

e 

learning.  Using  an  appropriate  estimate  for  a 

/N 

estimate  of  the  standard  error  for  dRLSR  is 


s.e.(dLLSR)  =  12  Est(c£2)/[N(U2+1)] 


In  summary,  to  use  the  LLSR  method  to  test  for 
one  need  only  compute  the  following  steps 


N 

2  (t, -t ) (y .  ) 
_  i=l  1 

'LLSR  N  _  _ 

E  (t.-t)2 
i=l  1 


S’ e’ ^dLLSR) 


12  Est  (o  ) 


N(N  +1 ) 


Test  procedure: 


Commute : 


Ho: 

d  < 

LLSR  ~ 

0 

Hl: 

dLLSR  > 

0 

A 

dLLSR-0 
s .  e . (d  ) 


y 

a  test  for 
2,  the 


learning, 


(2-32) 

(2-33) 


(2-34) 


If  tQ  >  ta>Xj_0  reject  Hq  otherwise  the  test  must  be  regarded 
as  inconclusive. 

Non-Linear  Method  feo  Test  for  Learning 

If  the  degree  of  non-linearity  in  a  particular  perfor¬ 
mance  curve  is  small  enough,  it  may  be  possible  to  examine 
the  slope  based  on  linear  techniques.  Measures  of  non¬ 
linearity  have  been  developed  by  Beale  [4]  which  indicate 
when  the  degree  of  non-linearity  in  a  non-linear  function  is 
small  enough  to  justify  approximations  using  linear  theory 
results.  A  review  of  Beales  procedure  follows. 

Consider  the  non-linear  model  u  *  f(x,ij)  where  e  is 
a  (pxl)  vector  of  parameters  and  x  is  a  vector  of  independent 
variables.  Given  n  independent  observations  on  the  response 
n  =  (y1,y2, . . . ,yn) ,  a  least  squares  estimate  of  the  parameters 
9  =  (e x ,  ...»  8  )  is  obtained.  Then  the  tangent  plane  approxi¬ 
mation  to  the  solution  locus,  estimation  space,  in  the 
neighborhood  of  9  is  given  by: 

x  P  -  9fCx.  ,0) 

x •  (9 )  =  n  •  (9)  +  £  (9.-0.3  I x  i  =  1,2,. . . , n  (2-35) 

i  i  j=i  J  J  ddl  0 

—  A  A 

or  t  (0)  =  n  (9)  +  x(0-e) 

7T  * 

where  n^(0)  =  f(x^,0).  Since  r(8)  differs  from  the  actual 
point  n(9)  because  of  the  non-linearity  of  equation  [1-2],  a 
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crude  measure  off  non-linearity  would  be 


% 


m 

Z 

w=  1 


n 

E 

i  =  l 


[ni(ew) 


*  p  -  sf(x. ,e)  7 

(ni(9)  -  .r  (ejw-ep  — Ip) 

(2-36) 


Qe 


m 

E 

w=l 


ln(ew) 


-C®„>l  I 


2 


Q  is  defined  by  Guttman  and  Meeter  [6]  as  the  sum  of  squares 
of  the  distances  (in  sample  space)  from  the  points  n(§w)  t0 
the  associated  points  t(0w)  on  the  tangent  plane.  By  dividing 

Q  by  the  quantity 

0 


m  n  a  n  ni  ~  a 

z  {I  [n.fej-HiO)]2}2  =  E  ||n(ew)-n(0)H 

w=l  i=l  w=l 


the  sum  of  squared  distances  is  normalized.  Guttman  and 
Meeter  [6]  go  on  to  explain  that  since  Qg  has  the  dimension 
of  the  square  of  an  observation  and  the  quantity  in  equation 
(2-37),  the  dimension  of  the  fourth  power  of  an  observation, 
then  the  quantity 

n0  =  ps2  z  1 1  n(ew)  -t(§  )  1 12/  z  I  |n(ew)-n(e)|  |4 
w=l  w=l 

2  2 

where  s  is  an  estimate  of  a  (the  variance  of  the  observa- 
tions) ,  is  a  dimensionless  quantity.  This  value  of  NQ 
can  be  regarded  as  the  estimated  normalized  measure  of  the 


non-linearity  of  the  model  when  expressed  in  terms  of  the 
parameters  0.  Beale  says  that  the  linear  approximation  is 
satisfactory  if 


N.  5  0.01/F 

0  a,p,N-p 


(2-39) 


since  the  root  mean  square  value  of  the  discrepancy  vector 

n(0w) -x(0  )  is  less  than  one-tenth  the  length  of  the  intended 

—  —  A 

vector  x ( 0  ) - n  ( 9) .  (See  Figure  2-9  below.) 


Intended  Distance 


Figure  (2-9).  Illustration  of  Discripancy  Distance 
Versus  Intended  Distance 


In  our  case,  if  the  degree  of  non  linearitv,  N  ,  of 
the  model,  equation  (2-3),  satisfies  inequality  (2-39), 
then  the  linear  theory  results,  with  an  appropriate  correction 
factor,  can  be  applied  to  find  the  confidence  region  for  the 
non  linear  performance  function  for  given  values  of  9  using 


SS ( 9) -SS(6)  =  ps  F 


(correction  factor) 


(2-40) 


a,p,M-p 

Conversely,  it  will  also  be  possible  to  find  the  confidence 
limits  on  the  parameters,  0  =  (a,b),  by  solving  equation 
(2-40)  when  the  confidence  region  for  the  function  is  speci¬ 
fied.  Going  one  step  further,  it  will  also  be  possible  then 
to  compute  the  confidence  limits  for  the  slope  of  the  curve, 
where 


slope  =  abt'(b+1)  (2-41) 

at  any  trial,  t^,  by  conducting  a  search  over  the  periphery 

of  the  joint  confidence  region  of  the  parameters. 

The  correction  factor  discussed  by  Beale  [4] ,  in 

2 

equation  (2-40)  when  p=2  and  s  is  replaced  by  SS(0)/(N-p)  is 


[1 


♦  NCP^2) 
(N-p)p 


V 


and  includes  the  measure  of  non-linearity,  N0 ,  which  allows 
for  the  effect  on  non  linearity  on  the  usual  linear  theory 
results . 

As  the  difference  between  n(0  )-t(0  ),  for  a  non¬ 
linear  function  (see  Figure  2-9)  increases  within  the  region 
where  the  inequality  sign  in  equation  (2-39)  does  not  change, 
the  allowable  size  for  the  variance  decreases.  Therefore  a 


restriction  using  linear  theory  approximation  is  that  the 
variance  might  have  to  be  unreasonably  small  for  the  approxi 
mations  to  hold. 

A  study  was  conducted  to  determine  the  maximum  size 
variance  allowable  for  varying  degrees  of  non-linearity  of 
the  performance  curve  function  for  a  particular  a  level  of 
significance.  As  the  value  of  the  coefficient  or  the  expo¬ 
nent  in  equation  (1-2)  increases,  the  maximum  allowable 
variance  decreases.  The  results  are  contained  in  tables 
(3-18)  and  (3-19). 

Procedure 

To  assist  in  following  the  non-linear  Drocedure  used 
to  test  for  learning,  an  overview  of  the  steps  required  is 
presented,  followed  by  a  detailed  discussion  in  Appendix  A. 

Step  1:  Estimate  the  parameters  of  the  performance 
function  and  the  variance  using  non-linear 
estimation  techniques. 

A 

Step  2:  Find  the  degree  of  non-linearity,  !1q  ,  for 
the  estimated  function  using  Beale's 
measure  of  non-linearity. 

.A 

i.  If  N„  <  .  01/F  ..  proceed  to  step  3 

0  a  .  p  ,  N  -p  1 

A 

ii.  If  N.  >  .01/F  ..  stoD,  following 

0  a,p,N-o 

procedure  not  valid 

Step  3:  Determine  the  confidence  limits  for  each 
Parameter  of  the  performance  function  by 
satisfying:  equation  (2-40)  using  a  direct 
techn ique . 
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Step  4:  Find  the  confidence  limits  for  the  slope  at 

any  particular  trial  using  a  direct  search 
technique  over  the  periphery  of  the  joint 
parameter  confidence  region.  The  maximum 
and  minimum  values  for  the  slope  will  be  the 
upper  and  lower  confidence  limits  respectively. 

Step  5:  Examine  the  confidence  limits  for  the  slope: 

(i)  If  the  confidence  interval  contains 
zero,  do  not  reject  the  hypothesis 
that  learning  did  not  occur  during  this 
trial . 

C i i )  If  the  confidence  interval  does  not 

contain  zero,  then  it  can  be  concluded 
that  learning  is  taking  place  during  this 
trial . 

Steps  1-4  are  explained  in  greater  detail  in  Appendix  C. 

A  computer  program  is  located  in  Appendix  D. 
which  estimates  the  parameters  and  the  variance,  tests  the 
degree  of  nonlinearity  and  computes  the  confidence  limits 
on  the  slope  at  any  particular  trial. 
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CHAPTER  III 

EVALUATION  OP  PROCEDURES 

Since  the  accuracy  of  the  test  on  the  slope  will  depend 

2 

on  the  quality  of  the  estimate  of  ,  a  computer  simulation 

2 

was  conducted  to  compare  the  estimators  of  presented 

in  Chapter  II.  By  varying  the  parameters  for  any  given 
sample  size  of  observations,  we  can  simulate  different 
situations  that  could  occur  in  an  actual  test. 

2 

Evaluation  of  the  Bias  in  Estimating  oc 
The  expected  bias  value  was  computed  first  for  each 
estimator  for  specified  parameter  values  and  sample  size. 

As  the  exponent  value,  b,  or  the  coefficient,  a,  increased, 
there  was  a  corresponding  increase  in  the  bias  value 
associated  with  each  estimator  (see  Figures  3-1  through 
3-6).  This  resulting  increase  in  bias  as  the  parameter 
values  are  increased,  corresponds  to  a  more  severe  lack  of 
fit  of  the  linear  model  to  the  actual  process.  The  best 
estimator  using  minimum  expected  bias  as  the  selection  criteria, 
is  equation  (2-8).  The  expected  bias  of  this  estimator 
for  given  parameter  values  and  sample  size  was  approximately 
40  percent  less  than  the  expected  bias  obtained  using  the 
next  best  estimator,  equation  (2-9).  The  expected  bias 
associated  with  equation  (2-7)  was  4  times  larger  than  the 
bias  factor  using  equation  (2-8). 


Bias 


.12 


Fipure  3- 


Vi 

C5 

•pH 


Figure  3-2.  Fxpected  Bias  Psing  Fquation  ( P.-8  ) 
to  Estimate  c?_2,  N  =  6 . 


.12 


Figure  3-3.  Fxpected  Bias  Psing  Equation  (  ,?-9  ) 
to  Estimate  <r  2  v  »  5  _ 


Figure  .3-4 


Figure  3-6 .  Expected  Bias  Using  Equation  (  2-9  ) 
to  Estimate  a  2,  N  =  15 


A  simulation  study  was  conducted  to  evaluate  the 

variance  of  each  estimator  for  all  combinations  of  sample 

size  'I  =  6,  15;  standard  deviation  0£  =  .03,  .05,  .07,  .00; 

and  parameter  values  a  =  .1,  .3,  .5  and  b  =  0,  .A,  .3,  1 . 0 . 

To  reduce  sampling  variability  in  the  evaluation,  the  same 

stream  of  normal  randomly  generated  observations  was  used 

for  each  estimator.  One  run  consisted  of  1000  experiments 

for  the  specified  sample  size,  parameter  values  and  variance 

The  results  obtained  are  in  terms  of  percent  of  the  extimate 

within  the  soecified  tolerance  limits  (a  ±6),  when  usin?  a 

e 

particular  estimator. 

Designating  any  estimate  as  good  or  bad  depending 
on  whether  it  falls  within  or  outside  the  tolerance  limits 
respectively,  then  the  generating  process  of  a  particular 
estimator  will  follow  a  binomial  distribution.  Using  th« 
worst  case  for  estimating  the  variance  of  the  process,  ( 1 . e . 
P  =  .5),  we  can  approximate  the  variance  as  . 25/N.  To 
obtain  a  95%  confidence  interval  for  the  percent  of  good 
extimates,  p,  with  limits  (p  ±  .03),  an  appropriate  sample 
size  would  be  calculated  as  follows: 

p  +  .03  =  p  +  2/^7N 


Letting  p 


.5 
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N  =  ^pq 

( • 03) Z 


N  =  1100 


The  number  of  experiments  for  each  combination  was  set  at 
1000.  The  actual  variance  for  p  may  be  less  for  a  given 
condition  since  p  =  .5  will  give  the  upper  value  of  the  vari¬ 
ance  for  this  process. 

Since  the  number  of  simulation  runs  is  sufficiently 
large,  N  =  1000,  a  normal  approximation  of  the  binomial 
variables  can  be  used  to  construct  a  significance  test  [ll]. 
To  determine  if  the  percent  of  good  estimates  using  one 
estimator  differs  from  the  percent  of  good  estimates  using 
another  estimator  conduct  the  following  test: 


V  P1  p2 


H1 :  p1  ¥  p2 


Comoute 


Z0  = 


pl-p2 


^(i-V 

N 


( l-$2 ) 
N 


(3-D 


If  Zn  >  Z  reject  Hn  and  assume  there  is  a  significant 
0  a/2  0  ° 

difference  in  the  p  values  for  the  two  estimators. 


The  actual  variance  of  the  difference  between  the 

P1  (1-Pj  )+Pp ( l-no  ) 

two  success  ratios  will  be  less  than  — — — ^ - —  due  to 

correlation  between  the  estimates  from  the  two  estimators. 
Although  the  assumption  of  independence  between  the  variances 
of  the  success  ratios,  p^  and  Pp,  reduces  the  power  of  the 
test,  we  can  gain  some  idea  of  the  significance  of  the  differ¬ 
ence  between  the  two  estimators  using  equation  (3-1). 

The  results  and  analysis  of  this  study  on  the  estima- 

2  < 

tors  of  a  follow.  When  the  true  variance  is  small  (a 
e  c 

.0036),  the  best  estimator  is  equation  (2-8),  regardless  of 
the  sample  size  or  parameter  values  of  the  curve.  As  the 
variance  increases  however,  equation  (2-9)  appears  to  perform 
better  under  certain  conditions  than  does  equation  (2-8).  See 
Figures  (3-7)  through  (3-1*0.  To  examine  this  situation 
further,  the  distribution  of  the  extimates  of  the  variance 
about  their  respective  expected  values  was  analyzed  for 
different  conditions.  The  effect  of  sample  size  appears  to 
be  the  same  on  each  estimator,  (i.e.  larger  N  results  in 
smaller  estimator  variance)  and  is  not  significant  in 
determining  the  best  estimator. 

A  general  description  of  the  distribution  of  the 
estimates  using  equation  (2-3)  and  equation  (2-9)  for  a  2- 
design  in  terms  of  a  and  oarameters  "a"  and  "b"  is  depicted 
in  Figures  3-15  through  3-22. 


The  results  noted  were: 


c,n 


N  -  15,  a  *  .30,  0£  =  .03,  6  =  .02 


OBS  represents  estimates  obtained  using  equation  (2-0 
SEX  represents  estimates  obtained  using  equation  (2-8) 
SER  represents  estimates  obtained  using  equation  (2-9) 

Figure  3-8.  Percent  of  Estimates  Within  the  Interval 
(ae+6)  when  N,  a  ,  a,  b,  and  6  are 
Specified 


-  "MBn  i 


N  =  6 ,  a 
6  =  .015 


OBS  SEX  SER 


DBS  SEX  SER 


OBS  SEX  SER 


OBS  represents  estimates  obtained  using  equation  (2-7) 

SEX  represents  estimates  obtained  using  equation  (2-8) 

SER  represents  estimates  obtained  using  equation  (2-9) 

Figure  3-9.  Percent  of  Estimates  Within  the  Interval 

(oE±6)  when  N,  a  ,  a,  b,  and  6  are 
Specified 
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S  SEX  SER 
b  =  .40 


99.5 


51.3 


OBS  SEX  SER 


N  =  15,  a  =  .50,  a£  =  .03,  6  =  .015 


34.2 


OBS  SEX  SER  OBS  SEX  SER 
b  =  .80  b  =  1.20 


85.5 


15,  a 
.025 


=  .50,  a 


35.2 


OBS  SEX  SER 
b  =  .80 


OBS  SEX  SER 
b  =  1.20 


OBS  represents  estimates  obtained  using  equation  (2-7) 
SEX  represents  estimates  obtained  using  equation  (2-8) 
SER  represents  estimates  obtained  using  equation  (2-9) 


Figure  3-10 


Percent  of  Estimates  Within  the  Interval 
(oe±6)  when  N,  a  ,  a,  b,  and  6  are 
Specified  f 
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OBS  SEX  SER  OBS-  SE 


07,  6 
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73.4 


59.3 


58.6 


28.9 


33.9 


OBS  SEX  SER 
b  =  .40 


OBS SEX  SER 


OBS  SEX  SER 
b  =  1.20 


OBS  represents  estimates  obtained  using  equations  (2-7) 
SEX  represents  estimates  obtained  using  equations  (2—8) 
SER  represents  estimates  obtained  uding  equations  (2-9) 


Figure 


Percent  of  Estimates  Within  the  Interval 
(ae±5)  When  N,  a  ,  a,  b,  and  6  are 
Specified 


f 


OBS  represents  estimates  obtained  using  equation  (2-7) 

SEX  represents  estimates  obtained  using  equation  (2-8) 

SER  represents  estimates  obtained  using  equation  (P-9) 

Figure  3-14.  Percent  of  Estimates  Within  the  Interval 

(ae±6)  when  N,  a  ,  a,  b,  and  6  are 
Specified 
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(i)  As  the  variance  of  the  process  increased  for 
large  values  of  parameter  b,  the  dispersion  of  the  estimates 
using  either  estimator  appears  to  be  a  heavy  tailed  normal 
distribution.  This  should  be  expected  since  larger  estimates 
of  the  variance  are  possible  which  will  weigh  the  expected 
value  of  the  estimates  to  the  right.  It  is  also  worthwhile  to 
point  out  that  when  parameter  "a"  is  small  in  this  situation, 
the  majority  of  the  estimates  obtained  were  less  than  the 
true  variance.  Thus  it  appears  that  near  the  asymptote,  when 
the  variance  is  large,  the  linear  model  approximation  using 
the  difference  method  results  in  a  "better  fit"  than  the  true 
model  in  terms  of  minimizing  the  sum  of  the  squared  errors. 

(ii)  When  the  variance  of  the  process  was  large  and 
both  parameters  small,  the  estimates  using  equation  (2-8) 
appeared  to  be  more  dispersed  about  their  expected  value  than 
the  estimates  using  equation  (2-9)  about  their  expected 
value.  When  the  expected  value  of  the  estimates  is  near  the 
true  value  of  the  variance,  a  tighter  distribution  of  the 
estimates  about  their  expected  value  can  result  in  more 
estimates  "ailing  within  the  tolerance  limits  (see  Figure 
i-17).  For  the  two  situations  discussed  in  (i)  and  (ii) 
above,  the  estimator  with  the  largest  expected  bias  generates 
•"''re  •*.-t ! mates  within  the  tolerance  limits  specified  for 

’  iiv.  T‘.  is  easy  to  see  in  Figures  1-17  and  3-13 
*  h.v  T  •’  •  h-  i  -ilepinco  limits  were  reduced  it  becomes  more 


i i  f  f  i x  1 1 


4  •>  i  i  ffereut  late  between  which  estimator  is  better. 


For  all  other  combinations  of  parameters  and  variance  size 
examined,  the  best  estimator  was  the  one  with  the  minimum 
expected  bias.  The  rule  for  selecting  which  estimator  of 


o£  is  better  when  the  parameters 
are  known  is: 

If: 


Ci) 

o  < 
e 

.06 

Use 

(ii) 

0  > 
e 

. 06 ,  a  - 

.5  Use 

(iii) 

a  > 
e 

. 06 ,  a  < 

.5  Use 

rnd  the  true  value  of  o 

e 

equation  (2-8)  to  estimate  a£ 
equation  (2-8)  to  estimate  o£ 
equation  (2-9)  to  estimate  a 


Since  the  true  value  of  the  parameters  and  the  actual  variance 
will  not  be  known,  we  must  base  our  rule  on  estimates  for 
and  parameter  "a".  A  general  rule  then  for  selecting  the 
best  estimator  when  a£  and  parameter  "a"  must  be  estimated 
is : 


Compute  an  estimate  of  0£  using  equation  (2-8) 


N-l 

o 

(N-l) 

E 

(x.-xr 

A 

rr  — 

i=l 

o  — 

E 

1 

2N(N- 

■i) 

(i)  0  <  .06 

£ 

A 

Use  0£  as  computed 

A 

A 

(ii)  o  >  .06 

E 

,  a  > 

.5 

Use  o^  as  computed 

(iii)  0£  >  .06 

A 

,  a  < 

.5 

Use  equation  (2-9)  to 

compute  an  estimate  o 

fectiveness  of  this  rule,  when  choosing  an  estimator 
%i!l  be  evaluated  in  conjunction  with  determining 


tr-‘ 


Figure  3-15.  General  Description  of  the  Distribution  of  the 
Estimates  of  ae  Using  Equations  (  2-8  )  and 
(2-9)  when  oe  is  Small,  a  is  Small,  b  is 
Small 


Figure  3-16.  General  Description  of  the  Distribution  of  the 
Estimates  of  oG  Using  F.ouations  (  2-8  )  and 
(  2-9)  when  o  is  Small,  a  is  Small,  b  is 
Large  c 


Figure  3-17.  General  Description  of  the  Distribution  of  the 
Estimates  of  ae  Using  Equations  (  ,?-R  )  and 
(7-9)  when  c  is  Large,  a  is  Small,  b  is 
Small  e 


Figure  3-18.  General  Description  of  the  Distribution  of  the 
Estimates  of  ac  Using  Equations  (  ’-8  )  and 
(7-9)  when  a  is  Large,  a  is  Small,  b  is 
Large 
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Figure  3-19.  Genera  1  Description  of  the  Distribution  of  the 
Estimates  of  oe  Using  Equations  (  ?-8  )  and 
(  2-9  )  when  a  is  Small,  a  is  Large,  b  is 
Small  e 
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Figure  3-20.  General  Description  of  the  Distribution  of  the 
Estimates  of  oe  Using  Equations  (  2-3)  and 
(  2-9)  when  a  is  Small,  a  is  Large,  b  is 
Large  £ 
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Figure  3-21.  General  Description  of  the  Distribution  of  the 
Estimates  of  ae  Using  Equations  (  2-8  )  and 
(2-9)  when  0  is  Large,  a  is  Large,  b  is 
Small  e 


Figure  3-22.  General  Description  of  the  Distribution  of  the 
Estimates  of  ae  Using  Equations  (  2-8  )  and 
(2-9)  when  a  is  Large,  a  is  Large,  b  is 
Large  e 
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the  best  test  procedure  later  in  this  chapter. 

Comparison  of  the  Linear  Test  Methods 
Test  Learning 

The  LLSR  estimate  of  the  average  rate  of  learning 
as  defined  by  equation  (2-23)  was  analyzed  in  terms  of  the 
percent  of  the  true  average  rate  of  learning  for  various 
combinations  of  sample  size  and  parameter  values.  If  the 
size  of  the  sample  is  increased,  the  amount  of  bias  in  the 
estimate  of  the  average  rate  of  learning,  equation  (2-20) 
using  the  LSSR  method  should  also  increase  in  a  negative 
direction.  Stated  another  way,  as  the  sample  size  increases, 
the  expected  value  of  the  LLSR  estimate  of  average  rate  of 
learning  decreases.  This  should  be  expected  since  the  observa 
tions  over  the  latter  trials  will  be  closer  to  the  asymptotic 
value  which  will  have  a  negative  effect  on  the  slope  of  the 
LLSR  model . 

As  can  be  seen  in  Figures  3-23  and  3-24  the  effect 
of  N  on  the  expected  value  of  the  average  slope  estimate 
using  the  LLSR  method  is  more  pronounced  than  its  effect  on 
the  ACD  slope  estimate,  which  is  unbiased. 


Performance  Performance 


In  Figure  3-24,  if  the  value  of  parameter  b  approaches 
zero,  the  ratio  of  the  LLSR  estimate  to  the  average  slope 
will  approach  1.  When  the  value  of  b  is  zero,  this  is 
equivalent  to  the  expected  values  of  the  observations  being 
at  the  asymptote  of  the  performance  curve.  We  showed  in 
Chapter  II  that  under  this  condition  the  LLSR  estimate  will 
be  unbiased.  Therefore  when  the  value  of  b  is  small,  the 
expected  value  of  the  LLSR  slope  estimate  will  be  very  close 
to  the  actual  average  rate  of  learning;  and  as  the  value  of 
b  increases,  the  expected  difference  between  the  LLSR 
estimate  and  the  true  average  slope  will  increase. 

The  effect  of  parameter  a  is  not  as  intuitively  obvious 
Since  parameter  "a"  can  be  factored  out  of  both  equation 
(2-23)  and  equation  (2-20),  we  see  that  the  expected  value 
of  the  LLSR  estimate  and  the  true  average  slope  change  by  an 
equal  multiple  factor.  Therefore,  although  the  LLSR 
estimate  and  the  actual  average  slope  values  change,  the 
ratio  of  the  two  values  is  not  affected.  This  ratio  is 
graphed  in  Figure  3-25  as  a  function  of  the  exponent  value,  b, 
and  sample  size,  N. 

To  evaluate  the  significance  of  the  bias  in  the  LLSR 
estimate  when  testing  for  learning,  it  will  be  necessary  to 
examine  the  effect  on  the  computed  test  statistic.  Recall 
that  the  test  statistic  for  the  LLSR  method  is 
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and  for  the  ACD  method  is 


Using  the  same  estimate  of  the  process  variance  when  computing 
the  slope  variance  for  the  LLSR  and  the  ACD  methods  respectively, 
a  comparison  of  the  expected  values  of  the  test  statistics 
was  made.  The  ratio  of  the  test  statistics  is  graphed  in 
Figure  3-26  as  a  function  of  the  sample  size,  N.  and  the 
exponential  value  b.  It  should  not  be  surprising  to  find 
that  when  the  value  of  b  is  small  for  a  particular  sample 
size,  the  ratio  of  the  LLSR  test  statistic  to  the  ACD  test 
statistic  is  greater  than  1.  This  is  because  the  ratio  of 
the  estimates  for  the  average  rate  of  learning  using  the 
LLSR  and  ACD  methods  is  greater  than  the  ratio  of  their 
respective  variances.  As  the  value  of  the  exponent,  b, 
increases  for  given  N,  the  ratio  of  the  average  slope  esti~ 
mates  becomes  smaller  and  the  ratio  of  the  test  statistics 
decrease . 

An  increase  in  the  sample  size  for  any  particular 


value  of  b  will  also  affect  the  ratio  of  the  test  statistics. 
This  is  hue  to  the  fact  that  both  the  LLSR  slop  estimate  and 
the  corresponding  standard  deviation  of  this  slope  estimate 
decrease  at  a  more  rapid  rate  than  the  AOD  slooe  estimate  and 
Its  corresponding  standard  deviation  for  increases  in  N  (see 
Figure(3-?6) .  If  the  ratio  of  the  slope  estimates,  LLSR  vs 
ACD,  decreases  more  rapidly  than  the  ratio  of  the  correspond¬ 
ing  standard  deviations  of  these  estimates,  then  when  N  and  b 
assume  certain  values,  the  AOD  test  procedure  will  become 
more  powerful.  Thus  if  we  knew  the  parameter  value  b,  we 
could  select  the  more  powerful  test  procedure  by  examining  the 
samel e  size  N.  Since  b  will  usually  be  unknown  and  difficult 
to  estimate  without  using  a  computer  search  technique,  we  need 
a  general  rule  based  only  on  the  sample  size. 

Recall  from  Chapter  I,  that  our  purpose  for  developing 
the  test  orocedures  was  to  test  if  learning  is  occurring 
during  an  evaluation  of  a  particular  system.  Since  the  people 
operating  the  system  have  undergone  extensive  training  prior 
to  the  evaluation,  it  seems  unlikely  that  the  rate  of  learning 
that  could  occur  during  the  trials  of  this  evaluation  would  be 
very  large.  Therefore  the  performance  curve  measuring  learnin 
during  an  evaluation  should  rarely  have  a  parameter  value  of 
b  >  1.0.  Based  on  this  consideration,  and  referring  back  to 
Figure  3-?6,  we  can  expect  the  best  linear  test  procedure  for 
detecting  learning  to  be  the  LLSR  method  since  the  expected 
value  of  the  test  statistic  for  b  <  1,0  is  greater  than  that 
for  the  ACD  method  when  N  <  25. 


Evaluation  of  Linear  Test  Procedures 

A  simulation  study  was  conducted  to  evaluate  the 
findings  in  the  previous  section  concerning  the  more  powerful 
test  procedure  for  all  combinations  of  sample  size,  N  =  6, 

N  =  15;  a  =  .03,  .05,  .07,  .09,  and  parameter  values 
a  =  .1,  .3,  .5  and  b  =0,  .4,  .8,  1.2.  The  test  procedure 
that  results  in  the  largest  percent  of  significant  test 
statistics  for  a  given  a  level  when  learning  is  actually 
occurring  will  be  selected  as  the  better  test.  The  study  was 
also  designed  to  evaluate  if  any  one  particular  method  of 
the  three  alternatives  considered  below  was  better  for 
estimating  . 

1.  Using  equation  (2-8  )  under  all  conditions 

2.  Use  equation  (2-9  )  under  all  conditions 

3.  Apply  the  rule  for  choosing  an  estimator  based 

on  estimates  of  o£  and  parameter  "a"  as  discussed 
in  the  previous  section. 

The  best  method  of  the  three  alternatives  will  be 
selected  based  on  the  largest  percent  of  runs  that  a  particular 
alternative  resulted  in  an  estimate  of  that  was  closest 
to  the  true  value.  One  thousand  simulation  runs  were  used 
for  each  combination  of  a£ ,  sample  size  and  parameter  values 
to  insure  that  the  true  percent  of  detection  of  learning  would 
be  within  ±3%  of  the  simulation  results  at  least  951  of  the 
time.  To  reduce  the  sampling  variability  in  the  evaluation. 


the  same  stream  of  normal  randomly  generated  observations  was 
used  in  evaluating  each  test  procedure . for  all  combinations 

of  a  ,  b  ,  a  ,  N . 

e 

The  results  of  the  first  part  of  the  simulation  study, 
the  evaluation  of  the  test  procedures,  are  given  in  Tables 
3-1  through  3-8.  The  labels  SEX,  SER,  and  RULE  correspond 
to  methods  1,  2,  and  3,  respectively,  used  to  obtain  an 
estimate  of  a£ .  Each  block  in  the  table  contains  the 
percent  of  times  the  LLSR  test  detected  learning,  t^;  and 
the  percent  of  times  the  ACD  test  detected  learning,  t^ , 
using  a  particular  method  for  estimating  .  The  test 
statistics  for  each  test  procedure  were  compared  at  the 
a  =  .05  level. 

The  results  substantiate  our  earlier  findings  that 
the  ratio  of  the  LLSR  test  statistic  to  the  ACD  test  statistic 
is  a  function  of  the  sample  size  and  parameter  b.  In  Figure 
3-26  we  found  that  for  values  of  b  6  1.0  and  sample  size  of 
N  =  6  and  N  =  15,  we  should  expect  the  LLSR  test  procedure 
to  be  more  powerful  than  the  ACD  test.  The  results  in 
Tables  3-1  through  3-8  support  this.  As  the  value  of  b 
increases  from  one  the  ACD  test  procedure  is  more  powerful 
and  the  results  also  verify  this.  The  type  I  error  appears 
to  be  larger  using  the  LLSR  test  procedure  than  the  type  I 
error  using  the  ACD  test  procedure.  This  is  expected  since 
the  LLSR  test  procedure  gets  more  powerful  in  comparison  to 
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the  ACD  test  procedure  as  b  decreases  (see  Figure  3-10).  Also 
note  however,  that  the  tvne  I  error  for-  the  LLSR  test  pro¬ 
cedure  is  greater  than  the  specified  a  level  but  decreases 
as  N  increases.  A  possible  explanation  for  this  is  that 
when  parameters  a  and  b  are  small,  the  estimates  of  a  are 

F. 

approximately  equally  distributed  about  c£  as  shown  in  ^icures 
3-15  and  3~17.  If  the  estimate  of  o  is  smaller  than  a  , 
which  is  the  case  over  50$  of  the  time,  we  increase  the  pro¬ 
bability  of  obtaining  a  type  I  error.  As  the  sample  size 
increases,  the  dispersion  of  the  estimates  of  about  the 
true  value  of  a£  becomes  tighter  and  we  have  fewer  estimates 
of  which  are  substantially  less  than  the  true  value  of  a  . 
thus  decreasing  the  probability  of  a  type  I  error.  When  the 
type  one  error  was  specified  at  a  =  .10  the  power  of  both 
test  procedures  increased.  See  Table  3-9. 

The  second  objective  of  the  simulation  was  to  determine 
if  one  particular  method  for  estimating  o  was  better  under 
certain  conditions  than  another.  The  results  are  shown  in 
Tables  3-10  through  3-17.  Using  either  equation  (2-8)  or 
equation  (2-9)  instead  of  applying  the  rule,  yielded  a  larger 
percent  of  better  estimates  for  almost  all  of  the  combinations 
examined.  It  appears  then,  that  the  estimates  of  a  and 
parameter  ’’a’’  are  not  accurate  enough  to  use  in  applvirig  our 
general  rule. 
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Table 


a= .  5 


a= .  3 


-1.  Percent  of  Significant  Tests  for  Learning  Using 
Three  Different  Methods  for  Choosing  an 
Estimate  of  oc  for  Given  Values  of 
a,  b,  N,  and  cre.  a  =■  .05 


N  = 

6 

o£  =  .03 

b  =  0 

b= .  4 

OC 

II 

b  =  l .  2 

SEX 

tR= . 069 
tD=.042 

tR= . 998 
tQ=. 995 

tR=l . 000 
tD=l. 000 

tR=. 999 
tD=i . 000 

SER 

tR= . 060 
tD=. 040 

tR= .994 
tD=. 982 

tR=l . 000 
tD=.999 

tR= . 998 
t p= . 996 

RULE 

tR= . 069 
tD=.042 

tR=. 997 
tD=. 994 

tR=1.000 
tD=l. 000 

tR=. 999 

V'997 

SEX 

tR=. 059 
tQ= .047 

tR». 913 
t  .  8  5  9 

tR=. 983 
tD= . 974 

tR=. 982 
tD= . 985 

SER 

tR=. 058 
tQ=.039 

tR= .875 
tD=.  813 

tR=. 940 
tQ= . 929 

tR= . 939 
tD=. 918 

RULE 

tR= . 059 
tD=.°47 

tR= . 914 

tp= . 862 

tR= . 9  73 
tp= . 966 

tR=. 959 
tD=. 950 

SEX 

tR= . 056 
tD=.043 

tR=. 318 
tD=. 272 

tR= . 467 
tD=.415 

t  R= . 4  9  7 
tD= .452 

SER 

tR=. 054 
tD=. 038 

tR= . 291 
tp= .256 

tR= . 442 
tD=. 395 

tR= . 446 
tD=.409 

tR= . 056 

tR= . 318 

tR= . 467 

tR=.497 

KULfc 

tD=. 043 

tD= .272 

tD= .415 

tD=. 452 

RULE 
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Table  3-2.  Percent  of  Significant  Tests  for  Learning  Using 
Three  Different  Methods  for  Choosing  an 
Estimate  of  oe  for  Given  Values  of 
a,  b,  N,  and  o  .  a  =  .05 


N  =6 


a  =  .  05 
£ 

b= .  8 


b=  1 .  2 


Table  3-3.  Percent  of  Significant  Tests  for  Learning  Using 
Three  Different  Methods  for  Choosing  an 
Estimate  of  a£  for  Given  Values  of 
a,  b,  N,  and  a  .  a  =  .05 

N  =  6  a  =  .07 

b=0  b= . 4  b= . 8  b=l . 2 


SEX 

tR=. 084 
tD=. 081 

tR=. 823 
tD=. 762 

tR=. 951 
tQ=. 933 

tR= . 968 
t  jj=  .  966 

a= .  5 

SER 

t  R=  .  0  8  5 
tD=. 078 

tR=.797 
tD=. 732 

tR=. 910 
tD=.898 

tR=. 933 
tD=.924 

RULE 

tR=. 083 
tD=.081 

tR= . 826 
tD=.764 

tR= . 934 
tD=. 918 

tR=. 948 
tjj= .  943 

SEX 

tR=. 077 
tD=. 069 

tR= . 506 
tD=.436 

tR= .731 
tD=.684 

tR=. 773 
tD= . 754 

a= .  3 

SER 

tR= . 078 

tp= . 066 

t  R=  •  4  9  9 
tD=.440 

tR= . 697 
t Q= . 636 

t  R= . 705 
tD=.699 

RULE 

tR=. 078 
t^= . 069 

tR=. 509 
tD=.45! 

tR= .705 
tR= . 649 

tR=. 708 
tD= . 704 

SEX 

tR= . 092 
tD=. 089 

tR= .197 
tD=. 159 

tR= . 246 
tD=. 221 

tR=. 249 
tD=. 225 

a= .  1 

SER 

t  R=  .  0  9  2 
tD=.079 

tR= . 186 
tD=.151 

tR= . 234 
tD=.214 

tR=. 238 
tD=. 222 

RULE 

tR=. 095 
tD=. 089 

tR= . 196 
t  .  16  0 

tR=. 241 
t^= . 160 

tR= . 241 
tD=. 219 

RULE 
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Table  3-4.  Percent  of  Significant  Tests  for  Learning  Using 
Three  Different  Methods  for  Choosing  an 
Estimate  of  oe  for  Given  Values  of 
a,  b,  N,  and  a  .  a  =  .05 


N  =  6  a  =  . 09 

e 

b=0  b= . 4  b= . 8  b=l . 2 


SEX 

tR= . 080 
tD=.074 

tR= .685 
t p= . 6  31 

tp= . 868 

tD=.843 

tp= . 886 

tQ= . 874 

a= .  5 

SER 

tR=. 083 
tD=.071 

tR= . 658 
tD=.594 

tR= . 8  34 
tD=. 799 

tR=. 831 
tD=. 831 

RULE 

tR= .077 
tD=. 073 

tR= . 680 
tD=.623 

tR=. 862 
tD=.834 

tR= . 860 
tpj= .  865 

SEX 

tR=. 089 
tD=.082 

tR=. 395 
tD=. 359 

tR= . 596 
tD=.545 

t  R=  .  5  9 1 
tD=. 582 

a= .  3 

SER 

tR= .082 
tD=. 078 

tR=. 395 
tQ= . 351 

tR= .581 
tD=. 523 

tR= . 5  58 
tD= . 548 

RULE 

tR= , 092 
tD=. 080 

tR= . 402 
tp= . 355 

tR=. 582 
tD=. 523 

tR= . 563 
tD= . 554 

SEX 

tR=. 079 
tD=.071 

tR= . 164 
tD= .138 

tR= .199 
tD=.169 

tR=. 235 
tD=.210 

a= .  1 

SER 

t  R=  .  0  7  8 
tD=.074 

tR= . 1 57 
tD=. 139 

tR=. 191 
tD= . 166 

tR= . 222 
tD=.210 

RULE 

tR=. 082 
tD=. 075 

tR= . 164 
tD= . 1 37 

tR= . 200 
tD=.168 

tR= .228 

tD=-217 

RULE 


86 


Table 


a= .  5 


a= .  3 


3-5.  Percent  of  Significant  Tests  for  Learning  Using 
Three  Different  Methods  for  Choosing  an 
Estimator  of  a  for  Given  a,  b,  N,  and  a  . 
a  =  .05  e  E 


N  =  15 

a  =  .03 
£ 

b=0 

b= .  4 

b= .  8 

6*1.2 

SEX 

tR=. 039 
tD=. 050 

tR= 1 . 000 
tD=i . 000 

tR=l. 000 
tD=l . 000 

tR=l. 000 
tD=i. 000 

SER 

tR=. 039 
tD=.039 

tR=l . 000 
tD=l. 000 

tR=l . 000 
tD=1.000 

tR=l. 000 

tD=i.ooo 

RULE 

tR=. 039 
tQ= . 050 

tR=l. 000 
tD=l . 000 

tR=l . 000 

tjj-1.000 

tR=1.000 

tD=i.ooo 

SEX 

tR= . 064 
tD= . 034 

tR=l. 000 
tD=. 993 

tR=l. 000 
tD=l. 000 

tR=1.000 
tD=l. 000 

SER 

t  R=  .  0  5  5 
tD=. 029 

t  R=  .  9  9  8 
tD= .985 

tR=l . 000 
tD=. 999 

tR=. 999 
tD=.999 

RULE 

tR= . 064 
tD=. 034 

tR=1.000 
tD=. 993 

tR=l . 000 
tD=l. 000 

tR=l . 000 
tD=i . 000 

SEX 

tR= . 060 

V-057 

tR= .625 
tD=.462 

tR= . 694 
tjj= .  596 

t  R=  .  6  3  7 

tp= . 660 

SER 

tR=  .056 
tD=.044 

tR= . 598 
tD=.431 

tR= .654 
t^= . 556 

tR= . 587 
t^= . 5  90 

tR= . 660 

tR=. 625 

tR= . 694 

tR= . 637 

RULE 

t  p* .057 

tD=. 462 

tD=. 596 

tR= . 660 

RULE 
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Table  3-6.  Percent  of  Significant  Tests  for  Learning  Using 

Three  Different  Methods  for  Choosing  an 

Estimator  of  cr  for  Given  a,  b,  N,  and 

a  .  a  =  . 05 
e 


N  = 

b=0 

15  a  = 

e 

b= .  4 

OO 

II 

l n  xj 
o 

bs:l .  2 

SEX 

tR= .062 
tjj= .  041 

tR=1.000 
tD= .997 

tR=1.000 

tD=.999 

tR=1.000 

tD=i.ooo 

a= .  5 

SER 

tR= .060 
tD= .033 

tR=  .997 
tD= .991 

tR= ‘ 999 
tD= .998 

tR= .996 

tD=i.ooo 

RULE 

tR= .062 
tD=.041 

tR=1.000 
tR= . 994 

tR= . 999 
tD= .999 

tR= . 999 

tD=i.ooo 

SEX 

tR= . 062 
tD=.041 

tR= . 948 

V‘817 

tR=  .976 
tR= . 961 

tR= . 966 
tp= .976 

a= .  3 

SER 

tR= .060 
tD=.045 

tR= .930 
tD=.793 

tR= .958 
tD= .922 

tR= .895 
tD= .926 

RULE 

tR= .062 
tD= . 044 

tR= .948 
tD=.815 

tR= .970 
tD=  .955 

tR= .931 
tD=.967 

SEX 

tR= . 053 
tp= .055 

tR= .327 
tD=.247 

tR= .361 
tD= .320 

tR= . 356 
tD= . 362 

a=  .  1 

SER 

tR= . 054 
tD=.046 

tR= .304 
tD=.216 

tR= .340 
tD=.297 

tR~ . 331 
tD=.315 

RULE 

tR= . 054 
tD=.054 

tR= .332 
tR=  .2  53 

tR= . 364 
tD= .326 

tR= .363 
tD= . 364 

RULE 


Table  3-7. 


Percent  of  Significant  Tests  for  Learning  Using 
Three  Different  Methods  for  Choosing  an 
Estimator  of  a  for  Given  a,  b,  N,  a  . 
a  =  .05 


N  =  15 


a  =  .07 
e 


b=  .  4 


b=  .8 


b=l .  2 


1 

1 


Table  3-8.  Percent  of  Significant  Tests  for  Learning  Using 
Three  Different  Methods  for  Choosing  an 
Estimator  of  a  for  Given  a,  b,  N,  and  a  . 
a  =  .05  e 

N=15  a£  =  .09 

b=0  b= .  4  b= . 8  b-1.2 
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Table  3-9.  Percent  of  Significant  Tests  for  Learning  Using 
Three  Different  Methods  for  Choosing  an 


Estimation 
a  =  .10 

N 

b= .  0 

of  a  for 

E 

=  6  a 

E 

b= .  4 

Given  a,  b,  N, 

=  .03 

b= .  8 

and  a  . 

e 

b=l .  2 

SEX 

tR=.110 
tQ= . 094 

tR=l . 000 
tr)=  .998 

tR=l . 000 
tD=i . 000 

tR=1.000 
tD=i. 000 

a= .  5 

SER 

tR=.105 
t-= . 090 

tR= .999 
tD= .997 

tR=1.000 

tD=i.ooo 

tR= .999 

tD=i.ooo 

RULE 

V.no 

V.094 

tR=1.000 
tD= .998 

tR=1.000 

tD=1.000 

tR=.999 

tD=1.000 

SEX 

tR= .108 
tD= .090 

tR=.974 

#952 

tR=  .998 
tp= . 995 

tR= . 998 
tp® .998 

a= .  3 

SER 

tR= . 109 
tD=  .089 

tR= . 963 
tD= .929 

tR=.997 

tD=.990 

tR= . 996 
tD= .993 

RULE 

tR= .109 
tD= .090 

tR= .972 
tD= . 952 

tR= .998 
tD=.993 

tR= .996 
tD=.997 

SEX 

tR= .095 
tD= .083 

tR= .450 
tD=.404 

tR= .649 
tD=.594 

tR=.641 

tD=.635 

a= .  1 

SER 

tR= .095 
tD=.075 

.445 

V-399 

tR= .619 
tQ=  .573 

tR= .607 
tn=  .  596 

RULE 

tR= . 095 
tD= .083 

tR= . 450 
V.404 

tR= . 649 
tQ=  .  594 

tR= . 641 
tp® .635 

RULE 
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The  primary  concern  is,  however,  to  select  the  method 
for  estimating  a£  that  will  result  in  the  most  powerful 
test  for  detecting  learning.  Comparing  the  test  procedure 
results  with  the  estimator  results  it  appears  that  using  the 
minimum  biased  estimate  of  a£  does  not  necessarily  result  in 
the  most  powerful  test  (i.e.  under  those  conditions  when 
either  equation  (2-9)  or  the  general  rule  presented  on  page 
68  provided  the  largest  percent  of  minimum  biased  estimates  of 
o_,  the  largest  percent  of  significant  test  statistics  is 
-  taJned  when  equation  (2-fi)  is  used  to  estimate 

Recall  that  when  either  parameter  "a"  or  "b"  is  small, 
(a  <  .1),  (b  5  .4),  the  observations  are  all  very  close  to 
the  asymptote.  Under  these  conditions  the  rate  of  learning 
is  very  small  and  the  corresponding  average  slope  is  small. 

If  is  large,  the  probability  of  detecting  learning  is 
small.  When  gets  smaller,  the  probability  of  detecting 
learning  increases.  This  same  idea  holds  for  estimates  of 
a  .  Examining  the  distribution  of  the  estimates  for  both 
estimators  in  Figures  3-15  through  3-22  it  appears  that 
equation  (2-8)  will  always  produce  the  larger  percent  of 

smaller  estimates  of  a  .  Therefore  using  estimates  of  a 

e  e 

generated  using  equation  ( 2-8 )  when  applying  the  LLSR  test 
procedure  for  any  combination  of  sample  size,  and 
parameter  values  results  in  a  more  powerful  test  for  learning. 


Table  3-10.  Results  for  Each  of  the  Three  Methods  for 

Choosing  an  Estimator  of  ae  in  Terms  of  Percent 
of  Minimum  Biased  Estimates  of  a 


a= .  5 


a®  .  3 


a=  .  1 


a= .  5 


N  -  6  0£ 

=  .03 

b  =  0 

b= .  4 

b= .  8 

b-1.2 

SEX= .445 

SEX®. 776 

SEX®. 988 

SEX®. 999 

SER= .555 

SER® . 224 

SER®. 012 

SER® .001 

RULE® .447 

RULE® .778 

RULE®. 897 

RULE® . 583 

SEX= .442 

SEX® . 548 

SEX®. 823 

SEX®. 948 

SER=  .558 

SER®. 452 

SER®. 177 

SER®. 052 

RULE® . 552 

RULE®. 564 

RULE® .775 

RULE® .735 

SEX® .455 

SEX®. 440 

SEX® .471 

SEX® . 514 

SER® .545 

SER®. 560 

SER® .529 

SER® . 406 

RULE® .461 

RULE® .459 

RULE®. 47 6 

RULE®  .5  26 

; - 1 1 .  Results  for  Each  of  the  Three  Methods  for 
Choosing  an  Estimator  of  oe  in  Terms  of 
Percent  of  Minimum  Biased  Estimates  of  a 

N  =  6  a 

E  =  *05 

b=  0 

b=  .4 

b=  .  8 

b-1.2 

SEX®  .576 

SEX® . 560 

SEX® . 794 

SEX®. 952 

SER®. 424 

SF.R®  .440 

SER® . 206 

SER®. 048 

RULE® .486 

RULE®. 584 

RULE® .667 

RULE®. 593 

SEX®. 471 

SEX® . 477 

SEX® . 569 

SEX®. 714 

SER® .528 

SER® . 523 

SER®. 431 

SER® . 286 

RULE®. 549 

RULE®. 520 

RULE®. 470 

RULE®. 397 

SEX®. 431 

SEX®. 435 

SEX®. 492 

SEX® .479 

SER® .569 

SER® . 565 

SER® . 518 

SER® . 521 

RULE®.  536 

RULE®. 548 

RULE® .519 

RULE® .533 

Table  3-12.  Results  for  Each  of  the  Three  Methods  for 
Choosing  an  Estimator  of  ae  in  Terms  of 
Percent  of  Minimum  Biased  Estimates  of  a 


N  =  6 
b= .  4 


a  =  .07 

E 


b= .  8 


b=  1  .  2 


SEX*. 543 

SEX=. 487 

SEX*. 741 

SEX*. 798 

a= .  5 

SER= .457 

SER=.  513 

SER*. 256 

SER*. 202 

) 

RULE= .473 

RULE=. 517 

RULE*. 539 

RULE*. 547 

SEX= . 4  39 

SEX*. 467 

SEX*. 530 

SEX*. 556 

a= .  3 

SER=. 561 

SER=. 533 

SER*. 470 

SER* . 444 

RULE*. 458 

RULE=. 443 

RULE*. 380 

RULE*. 31 7 

, 

SEX= .461 

SEX-. 438 

SEX*. 456 

SEX*. 452 

a= .  1 

SER-.539 

SER=. 562 

SER* . 544 

SER*. 548 

RULE* .448 

R(JLE=  .459 

RULE*. 465 

RULE*. 445 

! 

Table 

I 

3-13.  Results  for  Each  of  the  Three  Methods  for 
Choosing  an  Estimator  of  in  Texms  of 

Percent  of  Minimum  Biased  Estimates  of  a 

1 

N  =  6 

04 

o 

II 

OJ 

o 

! 

b  =  0 

II 

b=.  8 

b  =  l .  2 

SEX* .449 

SEX-. 498 

SEX*. 558 

SEX* . 668 

a= .  5 

SER*. 551 

SER=. 502 

SER*. 442 

SER*. 332 

RULE-. 443 

RULE* .473 

RULE*. 554 

RULE*. 601 

SEX*. 453 

SEX-. 467 

SEX* . 467 

SEX*. 504 

II 

SER= . 547 

SER-. 533 

SER*. 533 

SER* .496 

RULE* .492 

RULE*. 482 

RULE*. 438 

RULE*. 447 

SEX=. 440 
SER=. 560 
RULE-. 498 


a  = .  1 


SEX= .454 
SER*. 540 
RULE-. 472 


SEX=. 444 
SER=. 556 
RULE= . 494 


SEX= .477 
SER* . 552 
RULE=. 472 
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Table  3-14.  Results  for  Each  of  the  Three  Methods  for 
Choosing  an  Estimator  of  ae  in  Terms  of 
Percent  of  Minimum  Biased  Estimates  of  a£ 

N  -  15  a  =  .03 

£ 


C  . 
II 

S 

b= .  4 

b= .  8 

b=l .  2 

SEX= .426 

SEX=. 991 

SEX= .999 

SEX-1.000 

LO 

II 

03 

SER=. 574 

SER= .009 

SER=. 001 

SER=0. 00 

RULE= .426 

RULE=. 989 

RULE=. 995 

RULE=. 893 

SEX= .411 

SEX= . 788 

SEX=. 991 

SEX=. 999 

a= .  3 

SER=. 589 

SER=. 212 

SER= .009 

SER=. 001 

RULE= .411 

RULE= . 788 

RULE=. 992 

RULE=. 991 

SEX=. 397 

SEX= .418 

SEX=. 506 

SEX=. 555 

a= .  1 

SER=. 603 

SER=. 582 

SER= .494 

SER= .445 

RULE= . 397 

RULE= .445 

RULE=. 475 

RULE=. 572 

Table 

3-15.  Results  for  Each  of  the  Three  Methods  for 
Choosing  an  Estimator  of  a£  in  Terms  of 
Percent  of  Minimum  Biased  Estimates  of  o 

N  =  15  oe  =  .05 

b=0  b= . 4  b= . 8  b= 1 . 2 

SEX= .426 

SEX=. 784 

SEX=. 991 

SEX-1. 000 

P 

II 

Cn 

SER= . 5" 4 

SER=. 216 

SER=. 009 

SER-0.000 

RULE= .487 

RULE-. 734 

RULE=. 794 

RULE=. 644 

SEX= . 387 

SEX=. 509 

SEX=. 750 

SEX=. 896 

P 

II 

'^4 

SER= .613 

SER=. 491 

SER= .250 

SER=. 104 

RULE= .508 

RULE= . 555 

RULE= . 586 

RULE=. 542 

SEX= .501 

SEX= . 4  2  2 

SEX= . 4  36 

SEX=. 474 

a= .  1 

SER= . 4  99 

SER=. 578 

SER=. 564 

SER=. 526 

RULE= .501 

RULE= .490 

RULE= . 506 

RULE= .506 

SEE 


Table  3-16.  Results  for  Each  of  the  The  Three  Methods  for 
Choosing  an  Estimator  of  in  Terms  of 
Percent  of  Minimum  Biased  Estimates  of  a 


a= .  5 


SEX= .405 
SER* .595 
R(JLE=  .4  30 


N  =  15 
b=.  4 

SEX* .612 
SER* . 388 
RULE*. 526 


o  =  .  07 
£ 


b= .  8 

SEX=. 893 
SER=. 107 
RULE*. 568 


b=  1 . 2 


SEX= 

SER= 

RULE* 


.969 

.031 

.517 


SEX=. 434 

SEX*  ,46  7 

SEX= .597 

SEX= 

.  662 

SER* . 566 

SER*.  533 

SER= .403 

SER= 

.  338 

RULE= . 469 

RULE* . 370 

RULE=. 263 

RULE= 

.216 

SEX= .443 

SEX* .411 

SEX*. 440 

SEX*. 419 

SER= .557 

SER* .589 

SER*. 560 

SER*. 581 

Rl!LE=  .412 

RULE*. 449 

RULE*. 411 

RULE*.  394 

Table  3-17.  Results  fi-r  Each  of  the  Three  Methods  for 
Choosing  in  Estimator  of  oc  in  Terms  of 
Percent  of  Minimum  Biased  Estimates  of  a 

N  =  15 

a  =  . 
£ 

09 

b  =  0 

b= .  4 

b=  .  8 

b  =  l .  2 

SEX*. 427 

SEX*. 546 

SEX*. 736 

SEX*. 853 

a*. 5  SER*. 573 

SER*. 454 

SER*. 264 

SER*. 147 

RULE*. 490 

RULE*. 569 

RULE*. 644 

RULE*. 620 

SEX*. 422 

SEX*. 456 

SEX*. 474 

SEX*. 567 

a*. 3  SER*. 578 

SER*. 544 

SER*. 526 

SER*. 433 

RULE*. 553 

RULE*. 516 

RULE*. 478 

RULE*. 438 

SEX*. 429 

SEX*. 440 

SEX*. 440 

SEX*. 421 

SER*. 571 

SER*. 560 

SER*. 560 

SER*. 579 

RULE*. 526 

RULE*. 546 

RULE*. 526 

RULE*. 563 
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Evaluation  of  the  Nonlinear  Test  Procedure 

A  final  simulation  study  was  conducted  to  determine 
if  the  nonlinear  test  procedure  is  a  more  powerful  test  than 
the  linear  test  procedures.  The  study  was  conducted  under 
exactly  the  same  conditions  as  were  used  for  the  linear  test 
evaluation  (i.e.  same  normal  randomly  generated  observations 
were  used).  In  order  for  the  nonlinear  test  procedure  to 
yield  a  significant  test  statistic,  two  conditions  must  be 
satisfied : 

(i)  The  degree  of  nonlinearity  of  the  curve  must 
be  small  enough  to  apply  linear  theory 
approximation  for  determining  confidence 
limits  on  parameters  "a"  and  "b”. 

(ii)  The  lower  limit  for  both  parameter  "a"  and 
parameter  "b"  must  be  greater  than  zero  for 
the  lower  confidence  limit  of  the  slope 
abfb-l  to  be  greater  than  zero. 

An  advantage  of  the  nonlinear  test  over  the  linear 
test  procedure  is  that  it  is  possible  to  estimate  the  rate 
of  learning  at  any  particular  trial  by  examining  the 
confidence  limits  for  the  true  slope  at  that  trial.  Therefore 
if  the  degree  of  nonlinearity  is  small  enough,  less  than 
.01/F^,  fj_p »  ■*-*•  t>e  possible  to  construct  the  upper 

(1-a)  confidence  limit  for  the  true  rate  of  learning  at  any 
particular  trial.  If  the  value  of  the  upper  confidence 
limit,  in  terms  of  rate  of  learning  is  determined  to  be 
insignificant  at  a  particular  trial  then  we  can  assume  that 
learning  will  not  be  a  factor  in  any  future  trial  results. 
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As  can  be  seen  in  Table  3-18  when  either  parameter  "a" 
decreases  or  parameter  "b"  increases,  the  maximum  allowable 
o£  value  for  which  linear  theory  approximations  on  the  test 
of  the  slope  will  be  valid,  decreases.  For  the  nonlinear 
test  procedure,  the  results  in  Tables  3-20  and  3-21  indicate 
that  the  degree  of  nonlinearity  is  the  limiting  factor 
except  when  parameter  "a"  or  "b"  is  small.  An  explanation 
for  this  follows.  When  parameter  "b"  is  small  the  rate  of 
learning  is  small  and  the  curve  is  approximately  a  straight 
line.  A  tangent  line  approximation  for  estimating  the  values 
of  the  parameters  would  do  very  well  under  these  conditions. 
When  c£  is  large  in  comparison  to  the  average  rate  of  learning, 
it  will  reduce  the  power  of  the  test  before  it  becomes  a 
significant  factor  in  affecting  the  degree  of  nonlinearity. 

If  on  the  other  hand  the  value  of  parameter  "a’’  is  small,  this 
implies  that  the  observations  are  all  very  near  the  asymptotic 
value  of  1.  Since  we  can  never  do  any  better  than  100%, 
the  amount  of  deviation  above  the  expected  value  is  limited. 

If  o£  is  relatively  large  the  amount  of  deviation  that  occurs 
below  the  expected  value  may  well  exceed  the  limited  devi¬ 
ation  above  this  expected  value.  This  would  result  in  a 
fitted  curve  with  a  negative  value  for  b,  but  more  important 
it  will  also  result  in  a  lower  confidence  limit  for  the  slope 
which  includes  zero.  An  examination  of  the  observations  for 
several  simulation  runs  when  parameter  a  =  .1  and  o£  = 
did  reveal  this  to  be  the  case. 


.03 


Rl 


Table  3-18. 


a= .  5 


a= .  3 


a= .  1 


Maximum  Allowable  Value  of  oe  for  (iiven  Values 
of  "a"  and  "b"  for  Linear  Theory  Approximations 
to  be  Valid  at  a  =  .05  Level 

N  =  6 


b  =  0 . 0 

II 

b= .  8 

b-1.2 

00 

.078997 

.  042525 

.031026 

CO 

.047398 

.  025515 

.  018615 

00 

.  015799 

. 008505 

.006205 

Table  3-19. 


Maximum  Allowable  Value  of  ae  for  Given  Values 
of  "a"  and  "b"  for  Linear  Theory  Approximations 
to  be  Valid  at  a  =  .05  Level 

N  =  15 


a= .  1 

a= .  3 

a= .  5 


b  =  0 

b= .  4 

b= .  8 

b-1.2 

00 

. 073523 

.  035870 

. 025137 

00 

. 044114 

.  021523 

.  015022 

00 

. 014705 

.007174 

.  005027 
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Table  3-20.  Percent  of  Times  Linear  Approximations  can  be 
Used,  pr  .  and  the  Percent  of  Times  the  Slone 
Test  was  Significant,  Pgg 


b=0 

N  =  6 

b= .  4 

ae  =  -03 

b= .  8 

b-1  .  2 

PLA=1'000 

PLA=1.000 

PLA=' 895 

pla=.623 

a  = 

.  5 

pss 

.  080 

pss_1 

.  000 

PSS  '89S 

pss  ,623 

PLA=1 

.  000 

PLA= 

.  956 

PLA=-3" 

PLA=.200 

a  = 

.  3 

pss= 

.071 

pss= 

.  938 

pSS=.399 

Pss" *  20 U 

PLA= 

.  794 

PLA= 

.  267 

PLA= ’ 056 

PLA=-041 

a  = 

.  1 

pss= 

.053 

pss= 

.079 

pss=. 025 

pss=.025 

Table  3-21. 

Percent  of  Times  Linear 
Used  P^a  and  the  Percent 
Test  was  Significant  Pgg 

N  =  6 

b  =  0  b= .  4 

Approximations 
of  Times  the 

ae  ’°5 

b= .  8 

can  be 
Slope 

b-1 .  2 

PLA=1 

.000 

PLA=  -951 

PLA=  * 438 

PLA=-187 

a  =  .  5 

PSS= 

.080 

pss=  -940 

PSS= ' 438 

pss=-187 

PLA= 

.994 

PLA=  *579 

PLA=*104 

pLA=.056 

a  =  .3 

PSS= 

.072 

PSS=  *439 

PSS=* 103 

PSS= *056 

PLA= 

.145 

PLA=  -154 

pLa=.065 

PLa=.059 

a  =  .  1 

pss= 

.015 

PSS=  '°08 

PSS= • 005 

pss=-016 

100 


The  results  in  terms  of  the  percent  of  significant 
tests  per  1000  runs  using  the  nonlinear  test  procedure  are 
given  in  Tables  3-22  and  3-23.  As  the  value  of  a£  decreases, 
the  power  of  the  nonlinear  test  increases  when  the  values  of 
parameters  a  and  b  exceed  some  critical  number.  It  appears 
that  this  critical  value  may  be  near  .2.  As  the  sample  size 
increases  the  power  of  the  nonlinear  test  decreases.  This  is 
evident  by  comparing  values  in  Tables  3-18  and  3-19  since  the 
maximum  value  of  a  for  given  parameter  values  "a"  and  "b" 
decreases  as  N  increases. 

A  comparison  of  the  LLSR  test  procedure  with  the  non¬ 
linear  test  procedure  is  also  provided  in  Tables  (3-22)  and 
(3-23). 


Table  3-22 


a= .  5 


a=.  3 


a= .  1 


Table  3- 


a=  .  5 


a  = .  3 


a= .  1 
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Comparison  of  the  Percent  of  Significant  Tests 
for  Learning  Using  the  Nonlinear  Procedure, 
tNL»  ant*  the  LLSR  Procedure,  tp.  The  results 
are  based  on  1000  simulation  runs  for  each 
combination  of  a,  b,  N,  and  a£.  Tests  were 
conducted  at  a  =  .05  level. 


N  =  6 

c 

e 

II 

o 

04 

b  =  0 

b=  .  4 

b= .  8 

b=  1 . 

? 

tNL= 

.080 

tNL=1 

.  000 

tNL= 

.  895 

tNL= 

.623 

r-f 

70 

It 

.067 

*R  = 

.  998 

tR  =1 

.  000 

tR  =1 

.  000 

tNL  = 

.  071 

tNL= 

.  938 

tNL= 

.  399 

tNL= 

.  200 

tR  =, 

.  059 

tR 

.  913 

tR 

.  983 

tR 

.  982 

tNL= ' 

.  053 

tNL= 

.079 

tNL= 

.  025 

tNL= 

.  025 

tR 

.056 

tR 

.  318 

tR  = 

.415 

tR 

.497 

23.  Comparison  of  the  Percent  of  Significant  Tests 
for  Learning  Using  the  Nonlinear  Procedure, 
tNL»  and  the  LLSR  Procedure,  tp.  The  results 
are  based  on  1000  simulation  runs  for  each 


combination  of 

a,  b. 

N  and 

Oe-  ' 

Tests  were 

conducted 

at  the  a  = 

.05  level. 

N  = 

6 

°c  = 

.05 

b=  0 

b= . 

4 

b= . 

8 

b  =  l . 

2 

tNL=  * 

080 

tNL= 

.  940 

tNL= 

.438 

tNL= 

.187 

*R  =- 

086 

ZR  = 

.958 

tR 

.997 

lR 

.  995 

tNL=’ 

072 

tNL= 

.439 

tNL= 

.103 

tNL= 

.  056 

tR  =1 

080 

lR  = 

.715 

tR 

.912 

tR 

.894 

II 

-3 

Z 

015 

tNL= 

.  008 

Sl3 

.  005 

tNL= 

.  016 

rf 

70 

II 

084 

tR  = 

.  257 

tR 

.  328 

lR 

.  381 

CHAPTER  IV 


ILLUSTRATION  OF  THE  PROPOSED  PROCEDURE 

Example 

The  example  problem  chosen  was  an  actual  experiment 
conducted  at  Georgia  Tech  to  determine  the  Derformance  of  a 
viscous  damped  tripod.  The  experimental  results  were  in 
terms  of  standard  deviation  of  the  error  from  a  marked  point 
while  tracking  a  moving  target  at  a  constant  velocity.  The 
curve  describing  these  results  follows  the  form  of  equation 
(1-1).  Although  the  linear  test  for  learning  can  be  applied 
directly  to  the  learning  curve  data,  a  conversion  to  the 
performance  curve  will  be  made  in  order  to  apply  the  computer 
program  as  writtin  for  the  nonlinear  test  procedure. 

To  conduct  a  test  for  learning  using  the  performance 
curve  described  by  equation  (1-2),  a  suitable  scaling  of  the 
values  must  be  accomplished.  Recall  when  performance  Is  in 
terms  of  the  percent  of  total  possible  that  the  lower  limit 
at  trial  1  was  0  and  the  upper  limit  of  the  curve  at  some 
future  trial  number  was  1.  A  value  of  zero  Is  what  we  might 
expect  from  an  operator  who  is  totally  unfamiliar  with  the 
system  and  a  value  of  100%  is  the  expected  value  obtained  if 
the  system  meets  the  required  soecif ications  when  onerated  by 
a  fully  trained  individual  or  crew. 
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In  the  viscous  damped  tripod  experimental  data, 
values  that  correspond  to  the  lower  and  upper  limits  for  the 
performance  as  discussed  above  must  be  selected.  The  upper 
limit  will  be  chosen  as  the  minimum  standard  deviation  of 
error  predicted  by  the  manufacturer  when  the  tripod  is 
operated  by  a  fully  learned  individual.  The  value  selected 
to  correspond  to  the  lower  limit  on  the  performance  curve 
was  the  largest  standard  deviation  of  error  value  recorded 
during  the  first  two  trials  of  the  experiment.  The  tripod 
was  operated  by  5  different  individuals  whose  experience  in 
tracking  moving  targets  varied.  The  largest  standard 
deviation  of  error  over  the  first  two  trials  was  recorded 
by  a  subject  who  was  totally  naive  about  the  operation  of 
the  tripod  prior  to  this  experiment. 

The  equation  for  obtaining  the  proper  scaling  factor 
to  use  in  transforming  the  learning  curve  data  to  a  performance 
curve  is: 


where 

Em  represents  the  manufacturer’s  specifications 
Eg  represents  the  largest  value  for  standard  deviation 
of  error  recorded  during  the  first  two  trials 
Tp  =  transformation  factor  for  1  unit  of  change  in 


the  learning  curve  data. 
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The  interpretation  of  the  above  computations  is  that  ]  unit 
of  decrease  from  the  largest  value  recorded  in  the  learning 
curve  data  will  correspond  to  1 . 0/ [Eg -E^]  increase  from 
the  minimum  value,  zero,  on  the  performance  curve.  The 
equation  for  transforming  a  data  point  at  a  particular  trial 
on  the  learning  curve  to  the  performance  curve  is 


v  =  (Eg  -  trial  result)  (Tp)  (4-2) 

V  =  (6.522  -  trial  result#  . 18443 ) 

P 


where  Vp  represents  the  corresponding  value  on  the  performance 
curve . 

The  results  in  column  2  below  were  obtained  bv  an 
individual  who  was  familiar  with  tracking  moving  objects  but 
who  had  never  operated  this  particular  type  of  tripod  before 
this  experiment.  A  test  to  detect  if  learning  was  occurring 
during  the  first  6  trials  of  the  experiment  will  be  conducted 
using  both  the  LLSR  test  procedure  and  the  nonlinear  test 
Drocedure.  If  the  nonlinear  test  is  significant,  the  100 
(l-a)5?  confidence  limits  will  be  evaluated  for  the  slope  of 
the  curve  at  each  trial  to  determine  if  the  rate  of  learning 
becomes  insignificant  by  trial  6. 
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Trial 

Number 

Result 

Vyi 

X. 

l 

Cxi-X)2 

- t) 

1 

4.2124 

.4260 

.1328 

.  003564 

-1.0650 

2 

3.4920 

.  5588 

.0778 

.  000022 

-  .8382 

3 

3.0702 

.6366 

.1582 

.007242 

-  .3183 

4 

2.2126 

.  7948 

.0740 

. 000001 

.3974 

5 

1.8113 

.  8688 

-.0773 

.022620 

1.3032 

6 

2.2306 

.7915 

1.9789 

Applying  the  LLSR  test  procedure  at  a  =  .05  using  an  estimate 
of  from  equation  (2-8)  yields  the  following: 


H0:  dLLSR  -  0 


H0:  dLLSR  >  ° 


Compute : 


fc0  = 


dLLSR~° 


N ( N^+l ) 


If  tg  £  t  jj  do  not  reject  H( 


If  fc0  >  t.05,4 


rej  ect  Hr 


Compute  an  estimate  of  o'  using  equation  (2-8) 


(N-l)  E  (x.-x)2 
i=l  1 

2N(N  -  2) 


:  2  _  5(.  033449)  _  nn,t,0„. 

0 e  -  "TTTnrr  ~  * 0034843 


Compute  an  estimate  of  the  slope  using  the  LLSR  method: 


LLSR  = 


1f1Cti"Ejyi 

- -  2 

z  (t.-tr 

i=l  1 


H  1.4S8 

LLSR-  -1777 


dLLSR=  -0833 


Compute  the  test  statistic,  t 


dLLSR-0 


N (N^+l) 


*o  = 


.0833 


tQ  -  6.07 


Since  tQ  >  tpj  ^  we  reject  Hq  and  conclude  learning 
is  occurring  during  these  6  trials. 
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Applying  the  nonlinear  test  procedure  at  a  =  .05 
using  the  computer  program  in  Appendix  D,  yields  the 
following  results. 

Estimates  of  the  parameters  and  o£  are: 


a  =  .6017 
b  =  .6425 

A 

a  =  .0656 

E 

A. 

The  measure  of  non  linearity  N.  was  computed  as: 

y 


NQ  =  .00161373 

y 

Critical  Value  =  .001441 


The  degree  of  nonlinearity  for  this  estimated  curve  is  too 

large  to  use  linear  theory  approximations  to  compute  951 

confidence  limits  on  the  slope.  Applying  the  test  procedure 

at  a  =  .10,  the  test  for  learning  is  significant.  The  90? 

confidence  interval  on  the  slope  at  each  trial  is: 

Trial 

Number 


lower 

confidence 

value 

.1769 

’>pper 

confidence 

value 

=  .7366 

lower 

confidence 

value 

=  .0661 

upper 

confidence 

value 

=  .1902 

lower 

confidence 

value 

=  .0372 

upper 

confidence 

value 

=  .0878 

lower 

confidence 

value 

=  .0247 

upper 

confidence 

value 

=  .0518 
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5  lower  confidence  value  =  .0180 
upper  confidence  value  =  .0348 

6  lower  confidence  value  =  .0139 
upper  confidence  value  =  .0254 

This  particular  operator,  even  though  familiar  with  tracking 
targets,  still  appears  to  be  learning  after  6  trials. 
Therefore  we  would  conclude  that  none  of  the  6  trial  observa¬ 
tion  values  are  representative  of  the  performance  of  this 
tripod  when  operated  by  a  fully  learned  individual. 
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CHAPTER  V 

CONCLUSIONS  AND  RECOMMENDATIONS 

Conclusions 

This  research  involved  the  development  of  a  simple 
methodology  to  test  for  learning  in  experimental  results 
using  small  sample  sizes.  In  addition,  a  orocedure  for  exam¬ 
ining  the  instantaneous  rate  of  learning  at  any  particular 
trial  of  an  experiment  was  also  investigated. 

Assuming  learning  can  be  described  by  a  monotonically 

increasing  oerformance  curve  of  the  form  z  =  l-at-b,  tests 

for  learning  were  developed  based  on  examining  the  rate  of 

learning  over  several  trails.  Since  the  curve  is  monotonically 

increasing,  a  positive  slope  will  be  interpreted  as  learning 

and  a  zero  slope  will  correspond  to  no  learning  occurring. 

For  this  research  the  time  between  trials  was  considered 

insignificant  in  affecting  previously  gained  knowledge  and 

the  error  between  any  observation  and  its  expected  value,  z^, 

2 

is  assumed  to  be  NID  (0,  ). 

To  develoo  a  simple  methodology  to  test  for  learning  and 
to  provide  a  test  that  would  measure  the  rate  of  learning  at 
any  oarticular  trial,  required  two  different  approaches  to 
the  problem.  The  approach  to  develooing  a  simolified  test 
procedure  involved  examining  four  linear  methods  used  to 
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estimate  the  average  rate  of  learning  over  several  trials. 

A  comparison  of  the  variances  of  the  estimates  of  the  average 
rate  ^  learning  for  each  method  was  made  and  those  estimates 
with  minimum  variance  were  sleected  for  further  analysis. 

The  best  linear  method  was  the  linear  least  squares  regression, 
LLSR,  method  and  the  next  best  linear  method  was  the  average 
of  the  consecutive  differences,  ACD,  method.  Although  the 
variance  of  LLSR  estimate  of  the  average  rate  of  learning  is 

smaller  than  the  variance  of  the  ACD  method,  the  expected 

value  of  the  LLSR  estimate  contains  a  negative  bias  factor 
while  the  exoected  value  of  the  estimate  of  the  average 
of  learning  using  the  ACD  method  is  unbiased.  The  amount  of 

bias  in  the  LLSR  estimate  of  the  average  rate  of  learning 

increases  as  the  sample  size  or  the  initial  rate  of  learning 
increases.  When  the  ratio  of  the  LLSR  estimate  of  the  average 
rate  of  learning  to  the  ACD  estimate  of  average  ra^e  of 
learning  is  smaller  than  the  ratio  of  their  corresponding 
standard  deviations,  then  the  ACD  method  becomes  the  more 
powerful  test  procedure.  For  sample  sizes  less  that  24, 
the  LLSR  method  is  the  best  test  procedure  when  a  parameter 
b  is  less  than  or  equal  to  one. 

In  order  to  conduct  either  of  the  linear  test  procedures 

2 

an  estimate  of  the  process  variance,  ,  must  be  obtained. 

2 

lureo  estimators  of  a  were  examined  to  determine  which 

c 


2 

estimator  provided  estimators  of  a£  which  resulted  in  the 

most  powerful  test  for  learning.  These  estimators  were  derived 

2 

using  the  LLSR  estimate  of  o  ,  the  sum  of  the  squared  differ¬ 
ences  between  the  observations  and  pheir  average,  and  the  sum 

of  the  square  differneces  between  the  first  difference  of  the 

2 

observations  and  their  average.  The  estimator  of  o£  using 
the  first  difference  of  the  observations  provided  estimates 
which  resulted  in  the  most  powerful  tests  for  learning. 

Since  the  rate  of  learning  in  performance  evaluations 
is  not  expected  to  exceed  a  parameter  value  of  b  >  1,0,  the 
most  powerful  test  procedure  would  be  the  LLSR.  This  method 
is  most  powerful  when  the  variance  of  the  slope  estimate 

p 

is  computed  with  an  estimate  of  o£  obtained  using  the  first 
difference  estimator.  To  measure  the  rate  of  learning  at  any 
particular  trial  requires  an  analysis  of  the  instantaneous 
slope  of  the  curve  at  that  point.  The  approach  taken  was  to 
use  linear  theory  approximations  to  estimate  the  slope.  If 
the  degree  of  nonlinearity  of  the  function  is  small  enough, 
it  is  possible  to  use  a  linear  theory  approximation  to  con¬ 
struct  a  confidence  interval  for  the  true  slope  at  any 
particular  trial.  In  the  nonlinear  method,  estimates  for 
ae  and  the  parameters  "a'1  and  ’’b"  are  obtained  and  a  test  on 
the  degree  of  nonlinearity  of  the  function  is  conducted 
using  Beale’s  measure  of  nonlinearity.  If  the  degree  of  non¬ 
linearity  is  small  enough  then  it  is  possible  to  construct 
a  confidence  region  for  the  parameters  "a” 


and  ’b”.  By 


computing  the  estimate  of  the  slope  over  the  points  on  the 
periphery  of  the  parameter  confidence  region,  the  smallest 
and  the  largest  value  of  the  slope  estimate  can  be  obtained 
for  the  curve  at  a  given  trial.  The  smallest  and  largest 
value  of  the  slope  estimate  correspond  to  the  lower  and 
upper  confidence  limits  respectively  on  the  true  slope.  The 
width  of  the  confidence  interval  for  the  true  slooe  will 
decrease  over  the  latter  trials  until  it  will  eventually 
include  the  value  zero  which  indicates  a  fully  learned  status. 
Therefore  if  the  upper  confidence  limit  were  less  than  some 
maximum  acceptable  rate  of  learning  at  a  certain  trial,  then 
it  would  be  concluded  that  learning  would  not  be  a  factor  in 
any  future  trial  results. 

In  a  comparison  of  the  two  procedures,  the  linear 
methods  were  more  powerful  tests;  however,  the  nonlinear 
method  was  able  to  provide  information  on  the  rate  of  learning 
at  each  trial  when  the  nonlinearity  conditions  were  satisfied 
and  significant  learning  was  detected.  When  the  degree  of 
nonlinearity  was  small  enough  to  conduct  the  test,  signifi¬ 
cant  learning  was  detected  95?  of  the  time  except  when 
parameter  r'a"  and  "b'T  were  small.  The  more  powerful  linear 
test  procedure  was  the  LLSR  method,  which  <^an  detect  an 
average  rate  of  learning  over  15  trials  of  .01  at  an  a=.05 
level  95?  of  the  time  when  the  standard  deviation  is 
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The  advantages  of  the  linear  test  Drocedures  over  the 
nonlinear  procedure  are  greater  probability  of  detecting 
actual  learning  and  the  test  can  be  applied  using  simple 
arithmetic  and  comparing  computed  and  tabulated  values  as 
demonstrated  with  an  example  in  Chapter  IV. 

Recommendations  for  Future  Study 

In  order  to  apply  the  nonlinear  test  procedure  on  results 

which  follow  the  learning  curve,  equation  (1-1),  the  data 

must  be  transformed  into  data  that  follows  the  form  of  a 

performance  curve, equation  (1-2).  The  critical  factor  in 

the  transformation  of  data  is  selecting  a  value  from  the 

learning  curve  data  that  would  correspond  to  zero  performance 

on  the  performance  curve.  If  the  value  selected  is  too  small, 

the  scaling  factor,  T^ ,  will  be  inflated  and  when  the  value 

selected  is  too  large  T^  will  be  deflated.  The  effect  on 

the  power  of  the  test  procedures  is  not  clear  since  a 

2 

deflated  T_  results  in  a  larger  estimate  for  o  as  well  as 

r  t 

a  larger  estimate  for  the  average  rate  of  learning.  It  is 
recommended  that  future  research  be  conducted  to  investigate 
the  effect  of  errors  in  this  transformation. 

If  the  error  term  in  the  learning  curve  model  was 
multiplicative  rather  than  additive  as  assumed  in  this  study, 
the  test  procedures  developed  may  not  be  adequate  for 
detecting  learning.  It  may  be  necessary  to  develop  a  new 
test  procedure  based  on  the  logarithmic  transform  of  the  model, 
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ln(y)  =  ln(a)  +  bln(ti)  +  ln(ei) 
where  ln(y)  =  logarithm  (y) 

A  recommendation  for  future  research  would  be  to  develop  a 
methodology  for  detecting  learning  when  the  errors  are  log 
normally  distributed.  Assuming  that  lnCy^  is  normally 
distributed  then  ln(a)  and  b  will  be  normally  distributed; 
however,  the  distribution  of  eln^  =  a  is  not  normally  dis¬ 
tributed.  The  difficulty  then  in  developing  a  test  procedure 
for  this  apnroach  will  be  in  determining  the  distribution 
of  the  estimate  of  the  slope,  -abt~b_1. 

If  the  value  of  parameter  b  is  known,  it  would  be 
possible  to  select  the  test  procedure  which  would  be  more 
oowerful  for  detecting  learning.  It  is  known  that  if  the 
value  of  parameter  b  increases  for  a  given  sample  size  and 
and  a  given  value  for"a",  that  the  average  rate  of  learning 
also  increases.  If  an  estimate  of  the  value  of  "aM  is  made 
using  the  first  observation  and  the  value  of  M  is  known,  then 
it  may  be  possible  to  determine  the  value  of  b  by  examining 
the  estimate  of  the  average  rate  of  learning.  Another 
recommendation  for  future  research  is  to  study  the  relation¬ 
ship  between  the  average  rate  of  learning  and  parameter  b  in 
order  to  increase  the  probability  of  choosing  the  more  powers 
ful  test  when  the  true  performance  curve  is  unknown. 


<  ^ 
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APPENDIX  A 

EXPLANATION  OF  NOTATION 
Chapter  I 

",  value  of  the  true  oerformance  curve  at  trial  number  i 
x 

p 

a  variance  of  an  observation  about  its  expected  value 

e.  error  between  the  observation  and  its  expected  value  at 
1  trial  i 

Chapter  II 

y.  the  observation  at  trial  number  i 
i 

z  value  of  the  nerformance  curve  at  the  asvmtote 

c 

y  average  of  the  observations 

x.  the  difference  between  y^  +  1  and 

x  average  of  the  consecutive  differences  between  the 

observations 

S'  average  rate  of  learning 

/\ 

d estimate  of  the  average  rate  of  learning  using  the 
‘  average  of  consecutive  differences  (ACD)  method 

A. 

dT r „„  estimate  of  the  average  rate  of  learning  using  the 
‘  the  Linear  Lease  Squares  Regression  (LLSR)  method 

P  2 

sv^  minimum  variance  unbiased  estimate  of  a 

measure  of  degree  of  nonlinearity 

n . (0 )  true  value  of  the  function  at  trial  i  when  the  parameter 
values  are  0  =  ( 9^  , 8 ^ . 0? ) 

t  ^  ( 0* )  tangential  anoroximat ion  of  the  true  function  at  trial 
i  when  the  parameter  values  are  9  =  (0^ ,0^ , . , , ,9p) 

(OBS)^  estimate  of  a ^  using  equation  (2-7) 
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(SEX)?  estimate  of  a  ^  using  equation  (2-8) 

p  2 

(SER)‘  estimate  of  using  equation  (2-9) 

Chapter  III 

E[a  ]  expected  value  of  the  estimate  of  o£  using  the 
ex  average  of  consecutive  differences  estimator, 
equation  (  2-8  ) 

A 

E[a  ]  expected  value  of  the  estimate  of  0£  using  the 

LR  linear  least  squares  regression  estimator  (equation 
2-9) 

DBS  estimate  of  a£  using  equation  (2-7) 

SEX  estimate  of  using  equation  (2-8  ) 

SER  estimate  of  o£  using  equation  (2-9  ) 

RULE  estimate  of  a  applying  the  general  rule  described 

on  page  68  forF' choosing  an  estimator 

tR  percent  of  times  the  Linear  Least  Squares  Regression 

test  procedure  detected  significant  learning 

tj,  percent  of  times  the  Average  of  Consecutive  Differences 

test  procedure  detected  significant  learning 

P..  percent  of  times  that  the  degree  of  nonlinearity  of 

the  performance  curve  was  small  enough  to  use  linear 
theory  approximations 

percent  of  times  the  nonlinear  test  procedure 
detected  significant  learning  when  linear  theory 
approximat ions  could  be  applied. 

t  r  percent  of  time  the  nonlinear  test  procedure  detected 

significant  learning 

Tj.,  scaling  factor  for  transforming  learning  curve  data 

to  a  performance  curve 


largest  value  recorded  in  the  learning  curve  data 


APPENDIX  B 


DERIVATION  OF  THE  ESTIMATORS  OF  a 


Several  alternative  techniques  for  estimating  the 

p 

variance,  <7  ,  are  examined  in  terms  of  exoected  bias  to 

dtermirie  a  minimum  bias  estimator.  The  first  technique 
considered  will  be  that  presented  in  equation  (2-6), 


S  2  =  E  (y .  -y  )2/(N-l ) 
e  i=l 


( B-l 


writing  S  "  as  a  function  of  the  z^'s  and  taking  exoectations 
we  have 


E(s  =  E[  e  (y •  -y)/ (N-l) ] 
e  i=l  1 


=  ^  E{  E  (y  2  -  2yiy+y2) 
i=l 


1  N  ,  .f, N  (z  1 

■  5fTT[E  t  - ).(  r  »  -1112. 


!W>-  » 

-VW-***  • 

-  .  ■ 

Since  z^  is  a  constant  and  ~  N10  (0,o  “)  the  equation  in 
reduced  form  is 

N 

2  1  N  2  2  N  zi  f  Zi  N 

n(S  2)  =  E  (z-  +E(e.)  )  -2  E  ( - V - )-2  E 

i  =  1  i=l  i=l 

N  N  N 

E  , ,  E  z .  E  e  . 

•  i  l  N  .  ,  l  7  N  i  7 

r^iC-’-V—)]  +  E  E(i=^ — )  +  1  ] 

1=1  '  i=1  N 

■l  £  i 

Simplifying  where  E(e^2)  =  c^2  and  E(-~ — — )  ^  =  o^2/ N 


-  N  -  No  2  2  *  ZiZ 

E(Se2)  =  . E  ZiVN-l  ♦  -Vh:  *  —lTFY 

i  =  l 


in  _2  2 

0XI  2  /XT  E  z  No 
2N  °e  'N  i=l  e 

~Tl  +  in  +  NlNT) 


where  z  is  defined  as  the  average  of  the  expected  values  of 
the  observations. 


N  N 

E(S  2)  -  a  2  +  ^  [  E  z  • 2  -  2  E  z.z  +  Nz2] 
£  e  N-l  i  1 


crc  2,  2  1  t-  r,  ;,2 

E(se  )  °e  N.X  Izi'z] 
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Thus  the  farther  the  observations  or  trial  results  are 
from  the  asymptote  the  more  inflated  will  be  our  estimate  of 
the  true  error  variance.  A  candidate  for  consideration  as 
an  estimator  for  the  variance  then  is  equation  (B-l) . 

Another  simple  to  apply  method  for  estimating  the 
variance  of  the  process  would  be  to  use  the  first  difference 
of  the  series  of  observations 


Xi  =  yi+l 


-  y. 


where  is  the  observation  at  trial  i. 

Although  the  differences  are  not  independent,  the  variance 

2 

of  the  x.'s  will  contain  the  term  a  .  Therefore  it  mav 
x  e 

be  possible  to  express  the  variance  of  the  x^’s  as: 

A  Var  (x.  )  =  o^  +  B  (B-3) 

1  E 

where  A  defines  some  multiplier  and  B  repre-sents  the  bias 

factor  due  to  lack  of  fit  of  the  model  and  dependence  between 

the  values.  Then  a  comparison  of  estimators  can  be  made 

in  terms  of  expected  bias.  The  expected  bias  in  the  estimate 
2 

of  a£  using  the  sum  of  squared  errors  of  the  consecutive 
differences  between  observations  about  the  average 

?  N-!  ? 

S/  =  l  (x.-x)Z/N-2 
x  i  =  ]  1 


is  derived  below 


7  E(x. -x) 

H(Sx2)  -  El - — ] 


■  f/xi'x)  1  N-l  -I-, 

E[  Nl-y . i  =  nT2  E^i  =  1  ^yi  +  l"yi^  ’ 


N-l 

.Myi+rV 


N-l  -f  i  ^Zi  +  1  +  Ei  +  1')  '  ^zi  +  e  i  ■*  ' 

N-2  Et.E  r{(2.+1  +  ei+3)-(zi+ei)}-  ^ - TI - - — 1 

i  =  l 


N-2  E^  . l,  ^zi+l  +  £i  +  l‘zi"Gi'1  "  2^zi  +  l  +  Gi  +  l  zi  ei^ 
i  =  l 

N-l  N-l 

. 1 ,  ^Zi  +  l+ei  +  l ~zi  "ei  1  ^Zi+1+Ei+1 " "i  e  i  ^  0 

(ifi - ^ - )  ♦  (i=i - — — rn 


1  -  2 
E  (x. -x) 


'  1  '  ,  IN  - 1  2 

N^2  *  =  fTT  {E  i^1(zI=l+ei+l‘zi'Gi)  *2E  i^1 


(B-4) 
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1  N-l  } 


The  right  side  of  equation  (B-4)  may  be  considered  as  separate 
terms  and  the  expected  value  of  each  computed. 


«■:  "1  .»«v  *9''- 


First,  term: 


N-l  ?  -  N-l  ?  -  N-l 

i{  T  fz1  +  i  +  r  }  =  zl  +2  ,E  zi  +ZN  '2  2  zizi+l 

i = 1  i=l  i=l 


+  2  (N-l)o 


=  E  Czi+1-z  )2  +  2(N-l)<j  2 

i=  1  E 


Second  term: 


N-l 

.E,  ^i,rEin'zi-ei] 


ZN+£N~zl'£l- 


-2E{^E^  I  fzi  +  1  +  ei+l'zi'ei^  N-l 


zM+eM'Z1 -e1  N-l 
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zKI+eM- z,  -e , 
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ITT  F'{zN2+zl2'2zNZl+2zN£N'2zNEr2zleN'2eNel+2zlel+eN  T£1 


Taking  the  expected  value  of  each  quantity  inside  the 
parentheses  the  following  result  is  obtained 


2  f  2^  2  ~  ^  2v 

(Z\T  +Z1  ~2ZMZ-|  +  ^0  ) 


N-l  v  N  "1  N  1  ve 
,  2 

2  2  ^  ^  0 

in  ^n"2!^  m 


Third  term: 

N-l  2 

e  {  r  [  KTI  J  ' 

i-l 


NZ1  r  !;ZN+CN'Z1  +  E1)  t  2 

2  [ - ^ - j 

1  =  1 


1  N-i  2  2  2 

—  — T  s  fzN  +Z1  ‘2zNzl+2ae  ] 
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m —  Fnr 


Now  substitute  the  simplified  quantities  back  into  equation 
''5-4)  to  obtain 


E{ 


Z(x, 

i-l  x 

~FP7 


1 

N-2 


N-l 
{  Z 
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czi+rzi)2+2CN'1)ae2 


2 
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(zN-zx) 
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j-  2N  (N  -  2)  2 

{-nV~  °e 


N-l 

£ 
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The  LLSR  equation  for  estimating  the  variance  Is: 


o  n 


N  -  N 
MSp  =  {[  z  y. -  (  e  yj)/N 
i=l  i=l 


N 


.z  Cti-t)Cyi-y)  N 

2]  -  i — -  [  z  tiyi 
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“13 - 

I  (t.-t) 
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Compute  the  expected  bias  factor  due  to  lack  of  fit  of  the 
LLSR  model : 


E(MSe) 
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2  (t.-t) 
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Examine  each  term  inside  the  brackets  separately: 


pirst  term: 
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Third  term: 
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Replacing  the  quantities  in  equation  (B-7)  with  their 
expected  values : 
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Since  the  trials  are  consecutive  integer  values,  a  closed 
form  expression  for  t  in  terms  of  N  can  be  made  where 


y  t  -  N(N+1) 
i=l  1  Z 


;  .  2  _  N (N+l) (2N+1 
it-  -  - z - 


i-1  1 
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then  rewriting  equation  (B-8)  in  terms  of  N  where  applicable 
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Factoring  and  combining  like  terms: 
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Putting  this  in  the  form  of  equation  (2-8)  yields 
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Thus  another  estimator  for  the  variance  then  would  be 


S£2  =  (N2+l) (N-2)MSe/(N3-2N2+N+1) 

I 


(3-10) 
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APPENDIX  C 


NONLINEAR  TEST  PROCEDURE 


Estimation  of  Parameters  and  Variance 
Step  1.  Since  we  know  that  the  degree  of  nonlinearity 
for  the  performance  curve  function  is  small,  a  linearization 

method  for  estimating  the  parameters  should  work  quite  well 
[lO].  if  a  Taylor  series  expansion  of  the  function  f(tj,a,b) 
about  the  point  (a,b)  is  carried  out  and  curtailed  after  the 
first  derivative,  then  an  approximate  estimate  for  the  function 
f(t^,a,b)  is  then: 


fCq.a.b)  -  f(t1>Vb0)  *  (a-a0) 

ao 


+  ifcwo  (b.bo, 

bo 


fc-i) 


The  model  could  then  be  written  as: 


Y.  =  f(t.,a,b)  +  +  *  £i 


(C-2) 


To  solve  for  new  estimates  of  the  parameters,  minimize  the 
sum  of  squared  errors.  Let 

dl  = 

d2  =  (b-b0) 

6  =  (a,b) 


Ei2  -  [yj-fCVa^V  -  ^1^-1  (■>->„) 

ao 

-  3f(^aAI|  (b-bn)]2 


then  minimizing  with  respect  to  0  we  get 


9e,4  3f(t  ,0) 

-si —  -  2t-sr— ' 


3f(t,,e)  £(t.,e) 

— ^ — 'e  di  -  — 3f-'e„d2!) 


Ji3ijSI|0o  dl  -  2l£‘S\  d2l> 

Then  solve  the  two  equations  in  two  unknowns  for  the 
direction  vector  that  will  improve  the  initial  estimates  of 
the  parameters 


gf C 1 1 ,e )  n  3f(t.,0) 

m i — kn  di  +  -E,  d2 

0  1=1  0 


n  9f(t.,F) 

5®  l^o 


n  3f(t.,e)  n  3f(t.,0) 

ifi  ~dT^  U„  dl  +  J,  -3b  3b  Is  d2 


n  3f(t,,0) 
S  - - 


bv^W] 


The  minimum  distance  to  move  in  the  new  direction  D  = 
(d1,d2)  from  0q  =  (ao,bQ)  will  be 


0  =0  +  v  •  D 

new  o  mm 


(C-3) 


where  v  .  is  evaluated  as  follows: 
mm 

Compute  the  sum  of  squared  errors  for 


ss(ev)  =  s [y±-f Ct ,evD ] 


where  0  =  0  +  vD 


and  v  =  0,  1/2,  1 


Let  Q  (v)  =  SS(0v) 


then 


vmin  =  1/2  +  1/4  [Q(0)-QCi)]/[QC1)-2Q(1/2)+Q(0)] 


Using  the  new  estimate  of  the  parameter  vector,  0  w,  in 
equation  (C-3)  as  the  next  starting  value,  begin  another 


iteration . 


•  *  “Htf  I" 


When  the  value  of  D  =  0,  or  some  small  Incremental 
value,  6  =  .000001,  the  best  estimate  is  obtained  for  the 
parameters  that  minimizes  the  sum  of  squared  errors.  The 
corresponding  variance  would  be 


2  SS(0i_1) 


( C-4 ) 


where  9.  ^  represents  the  parameter  values  for  the  i"' 
Iteration . 

Step  2.  Determine  degree  of  nonlinearity.  The 
procedure,  as  discussed  earlier,  entails  using  Beale’s 
measure  of  nonlinearity  to  determine  if  the  degree  of  non¬ 
linearity  of  the  performance  function  is  less  than  .01/ 

F  ...  To  compute  the  measure  of  nonlinearity,  , 
a,p,N-p  r  1  0 

the  function  is  evaluated  along  with  the  corresponding  tangent 
plane  approximation,  equation  (3-34),  at  points  9w  =  (a,b) 

A 

in  the  neighborhood  of  0  =  (a,b).  Since  there  are  two 

parameters,  a  reasonable  design  for  considering  points  in 

2 

the  neighborhood  of  0  would  be  a  3  design.  In  order  to  keep 

7. 

the  distance  between  0  ,  w  =  1,  2,  3,  ...»  8  and  0  in 

7\  A 

proportion  to  the  size  of  0  as  0  changes,  compute  the  upper 

n 

and  lower  values  of  the  parameters  in  the  3*"  design  as 

A  /\ 

a  ±  1(a);  b  ±  %(b)  for  each  parameter  respectively.  A 
reasonable  percentage  value  would  be  between  3°  and  10%. 


Step  3.  Confidence  interval  for  parameters.  If  the 
degree  of  nonlinearity  satisfies  inequality  (3-38)  then  an 


approximation  for  the  linear  theory  confidence  region 


ssre)-ss(e) 


<  r  r ss  ( 0)  ■>  p  n 

-  p[TfrpTjFa,p,N-p’[1 


N(P+2) 
(N'-o)p  0 


( C  -  5 ) 


can  be  used  to  find  the  confidence  interval  for  each  parameter. 
A  direct  search  procedure  can  be  used  to  find  the  end  points 
of  the  confidence  interval  for  each  parameter  as  follows: 

(i)  Holding  parameter  "a"  at  its  least  squares 

estimated  value,  search  along  the  axis  of  "b" 
in  both  directions  from  b  until  equation  (C-5) 
is  satisfied.  The  smallest  value  for  b  and  the 
largest  value  for  b  that  satisfy  equation 
(C-5)  are  the  lower  and  upper  confidence  limits 
respectively  for  b. 

(ii)  Repeat  procedure  i,  reversing  the  roles  for 
the  parameters. 

Step  4.  Confidence  interval  for  the  slope.  If  the 
joint  confidence  region  for  the  parameters  can  be  defined, 
then  a  confidence  interval  for  the  slope  can  be  determined. 
Having  determined  the  confidence  interval  for  at  least 
one  parameter,  say  parameter  "b".  we  can  conduct  a  search 
at  incremental  noints  within  this  interval  along  the  parameter 


i 
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axis  to  obtain  peripheral  points  of  the  joint  parameter 
confidence  region.  The  value  of  the  slope  for  a  given  trial 
at  any  peripheral  point  corresponds  to  either  an  upper  or 
lower  limit  for  the  slope  at  those  particular  values  of 
(a,b).  To  determine  the  lower  and  upper  limits  of  the  slope 
at  a  given  trial  for  the  entire  joint  confidence  region, 
select  the  smallest  and  largest  value  of  the  slope  computed 
over  the  various  periphery  points.  The  smallest  and  the 
largest  slope  value  correspond  to  the  lower  and  upper 
confidence  limits  respectively  for  the  slope  at  that  given 
trial . 

Appendix  D  consists  of  a  computer  program  which  estimates 
the  parameters  and  a  ,  tests  the  degree  of  nonlinearity  and 
computes  the  confidence  limits  for  the  slope  at  time  t^. 
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APPENDIX  D 


COMPUTER  PROGRAM 
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D  J  MEN  8 1 ON  TIME'  ( SO  )  *  VE  (  10 * 20  )  *  ALPHA  i.  1 0  )  *  BETA  >  1. 0  >  •>  i  <  20  )  >• 
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0  I.  0  :l.  I  (  DBS  (  L  >  <  :l  .  O-PARMA  /  <  T I  ME  <  I  >  *  *  I'1  A  R’  M  B  >  )  ./  :*L  I. 

02  -0:2  L  <  OBS  (  I  >  -  (  1.  ♦  0 --PARMA  /  (  T IMF  ( I  )  #*PARMB  )  >  >  1 17  ! 
t  ONI  I  NHL 


SOLVE  FOR  ELEMENTS  UED I  RECTI  ON  VECTOR  II-IA I  LEI  L.L 
i:  H  P  R  0  V  E  OUR  E  S  1  I M  A  T  E  S  0  E  p  A  R  A  M  E  T  E  R  8  11  A  "  A  N  D  "  B  11 
ETNA  "Dl"  AND  "D2"  BY  SOLVING  A  2X8  MATRIX, 

1 1  (  I  I  I ,01,  l-P  GO  TO  13 

El  ISO-  I  I  0*H 
01  5  010*1-1 
IF  < F22U  , 


GT.  H)  GO  TO  14 
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F225:::F'225#H 
025- 02 5 *H 

III.  “0  l  5 

D 2=025 

FIND  MAXIMUM  HI STANCE*  VMIN  *  TO  PROCEED  IN  NEW 
DIRECTION  FROM  CURRENT  PARAMETER  VECTOR  10  CAIN 
AN  IMPROVEMEN  T  IN  MINIM t ZJ  NO  SUM  OF  SQUARED  ERRORS 


W  ••  i  *  0 
A  I -PARMA 
DJ.  • PARMB 

A  2  --  F'  A  R  M  A  +  (  W  *  ,  5  >  *  D  I 
B  2  ::P  A  R  M  .8  \  (  W  *  .  0  >  *  0  2 
A3  PARMA  IUIKDI 
8  3 FARM  8 + W  *  D  2 
0  A  I  0  A  2  "  0  A  3  =  0 . 0 
DO  1.5  I  •  1  * N 

0  A  I.  ~  0  A  :l.  F  (  0  B  S  <  I  >  ••••  (  I.  *  0  -  A  :l  /  (  T I M  E  <  I  >  *  *  8 1  )  )  )  *  *  2 

0A2  QA2  F  <  DBS  <  I  >  ••••  <  .1 . 0-  A  2/  (  T  I  ME  ( I  **.82  )  )  )  **2 

GA3-QA3+  (  DBS  (  I  )  ■-  (  I  ,  0-A3/  T I  ME  <  I  >  **B3  )  >  >  **2 

CONTINUE 

VAI...  1.  “0AI+QA3 

VAI...2  ■-•2.0*0A2 

IF  (VAI...  I  ,  EO  >  VAI...  2)  GO  TO  18 
V  M  I  N = 0  -.  5  F ,  2  5  *  <  0  A I  0  A  3  >  /  <  0  A  3  -  2  .  0  *  0  A  2 1 Q  A  .1.  > 

A  V =  P  A  R  M  A  +  V  M I N  *  W  *  D  I 
8  V  F‘  A  R  M  B + V  M I N  *  W  *  D  2 
BVU-0  .  0-5 , 0 

IF  >»V  , or,  BVV )  GO  TO  2 
GO  TO  16 
0V--0 .» 0 
DO  19  1=1 *N 

o  v = o  v  +  ( o  b  s  <  :i:  >  -  <  i .  o  -  a  v  /  ( r  i  m  e  <  :i: )  *  *  b  v  )  >  >  *  *  ? 

CONTINUE 
VAI.  •••  O'.1 -OAI 

IF  (VAI  •  I.  T  *  *00001 )  GO  TO  20 
W=W*  ♦  5 

WR I TE ( 6 . 97 >  qu 

FORMAT  (  ”  ”  /  11 Q  V  '  •  "  « I  I  I  *8) 

GO  !  0  6 :.v.i 
Dll  D  I 

D22--02 

IF’ (Dll  .  GT*  0*0)  GO  TO  61 
D  l  I.  (0,0  |  ,  0)*D  |  I. 

IF  (D  l.  I  *GT*  .000001)  GO  TO  62 
IF  (II 2 2  GT.  0*0)  GO  TO  63 
D22  :  (  0 . 0  1 . 0  )  *1)22 
I  F  (  D  2  2  , 1 .  T  *  .  0 0 0 0 0  I  >  G  0  T  0  1 6 


IF  (PARMBB  . EQ .  FARMS)  GO  TO  1 6 
6  2  P  A  R  M  A  =  P  A  R  M  A  +  0  M I N  *  U  *  D 1 
P  A  R  M  B = F;'  A  R  M  B  +  0  M I N  *  W  *  0  2 
PARMBB-PARMB 

IF  ( PARMB  ,LT.  5.0)  GO  TO  11 
GO  TO  16 
11  CONTINUE 

1 6  S  ==  <  0 A 1  /  ( N -■  2  )  >  **0 . 5 

W  R I T  e:  (  6  »  1 7  )  P  A  R  M  A  .  P  A  R  M  B  »  S 

1 7  F  0  R  M  A  T  <  11  "  /  "  P  AR  M  A  “  »  F 1 1 . 8  ?5X«  “  P  A  R  M  B  -  “  « 

C  F:'  1 1 , 8 . 5  X  y  "  S  T  D  D  E  M  =  "  ,  F- 1 1 , 8 ) 

CHECK  DEGREE  OF  NON-LINEARITY  USING  BEALE " B  METHOD 

SNUM  0 , 0 
SHE  H  O . 0 

GENERATE  COMBINATIONS  OF  THE  PARAMETERS  ABOUT  THE 
ESTIMATED  0 ALLIES 

B I  PARMB -0,08* PARMB 
DO  21  I - 1 »  9 

ALPHA < I ) -PARMA-0 , 08*PARMA 
B  E T A ( I )  • B 1 + ( I - 1 >*0< 0 8  *  P  A  R M  B 
II  (I  .LE.  3)  GO  TO  21 
A  I...  P  H  A  (  I  >  ~  P  A  R  M  A  +  0  ♦  0  8  *  P  A  R  M  A 
BE  T A  ( I  >  --B 1  +  (  I  -4  )  *0 , 08*PARMB 
IF  (I  .LE.  6)  GO  TO  21 
Al.  PH  A  <  1 )  -PARMA 
IF  (I  .EQ.  8)  GO  TO  24 
BE  TA  <  I )  -B 1  +  (  I  -7  >  *0 , 08*1"' All' MB 
GO  TO  29 

2  4  B  E  I  A  ( I )  --  B 1  +2.0*0. 0  8  *  p  A  R  M  B 
29  IF  U.  .  L..E .  8)  GO  TO  21 

BETA ( 1 ) -PARMB 
21  CONTINUE 

TEST  DEGREE  OF  NON-LINEARITY 

DO  22  I-:l  ,9 
DO  23  J--1 » N 

OF  (  I .  J  )  =  1 , 0  A  I...  P II A  (  I )  /  <  T I M  E  (  J  >  *  *  B  E  T  A  (  I )  ) 

3  CONTINUE 
2  CONTINUE 

DO  25  I 1  r  8 
DEM-  0 , 0 
DO  26  J-l.N 

X  :;0 . 0- 1 , 0  /  (  T I  ME  (  J  )  **BETA  (  9  )  ) 

Y  ■:=  0  »  0  ••••  A  l.  P  H  A  ( 9  )  *  A I...  0  G  ( 1 , 0  /  T  I M  E  (,.!>>*<<  J  .  0  /  T I M  E  ( ,J  >  )  *  *  B  E  T  r 
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(  (9)  > 

Q==  <  UE  <  I .  J  )  -  <  UE  (  9  -  J  )  f  (  ALPHA  (  I  >  A I  E  lift  BE  (A  i  I  > 

C--BETA  <  9 )  >*Y>  >**2 
SNUh-SNUM+O 

J.|  :  <  UE  ( I  -  J )  -UE  (  9  .  J  )  )  **2 
DEM- DEM  Hi 
26  CONTINUE 

S  D  E  M  • B  D  E  M I D  E  M  *  *  2 
25  CON  I'  INUE 

RE  BUI.  T~P*  ( 8**2 ) *  <  SNUM/SOEM  ) 

Ul  R I  I  E  <  6  f  2  7)  S  N  U  M  •  S  D  E  M  .  R  T  B  U I ...  T 

2 7  F 0 R M A  I  <  "  "  y  "  BNIIM  ~  ".Ell.  8  y  5 X  .  “  8 D E M  -  "  v  E  :l  .1  -By’ 'i X 

C "RESUI  I  -  “ y  E 1 0 , 8 ) 

DETERMINE  IE  LINEAR  THEORY  RESULTS  CAN  BE  USED  AS  nEEl- 
I AM A I  IONS  FOR  PARAMETERS  "A"  AND  "B" 

cpyAL.--o.oi  •  E 

IE  (RESULT  .  i  T  ,  CPOAI.  '•  00  TO  Li 
WRIT  E.  (  6  y  28  > 

28  FORMAT!  "  “."DECREE  OF  NON-LINEARITY  i'  00  I  AROF  “  • 

00  TO  60 

CO  Hr  UTE  THE  CONFIDENCE  LIMITS  FOR  THE  SI.. DEE  AT 
SPECIE  IF D  TIME 


COMPUTE  THE  COMPARISON  OALUE  FOR  THE  CONFIDENCE  I ES 

3  !  LICE- P*  C  8**2 )  *E*  ( I  .  O  f  <  ( N*  ( P+2 . 0  >  >  /  <  P*  •  N---P )  ■  i  iKESUi  i  > 
8  8  E  --  (  S  *  *  2  >  *  C  N  -  E’ ) 

D  £  T  E  R  M I N  E  I...  0  W  E  R  A  N  D  U  P  P  E  R  B  0 1 1 N  D  S  0  N  P  A  R  A  M  E  T  EE  “  b  "  ■ 
THEN  CONDUCT  A  SEARCH  ALONG  THIS  INTERVAL  TO  DETER¬ 
MINE  UPPER  AND  LOWER  BOUNDS  ON  THE  SLOPE. 

RED"=0 . 0005 
BL.  ™ PAR MB 
34  BL--BL -RED 
PAR  J"::0 , 0 
DO  32  I  •••■•  1  .  N 

CBL-  .1.  .  0  -PARMA/  (  TIME  ( I )  **BL  ) 

8 E  DI..  (  OB 8  C I  )  -  CBI... )  **2 
PARJ-PARJESPBL 

32  CONTINUE 

DIF  FBI...  -PARJ-SSE 

IE  (DIEEBL  . GE .  UCI  ) GO  TO  33 

GO  TO  34 

33  WRITE! 6. 35)  BE 


/"Bl.- 


.  F  I  I  ,  B ) 
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35  FORM  AH"  " 
BU-PARMB 
38  BU- BUT RED 
PARK-0 , 0 


DO  36  ]  1  *  N 

GDI.)  1 , 0 --PARMA  /  (  T I  ME  ( 1  )  **BU ) 
SPBU-  (  ODS  <  I )  -CBU )  **2 
PARK=PARK+SPBU 

36  CONTINUE 
DIFFBU-PARK-SSE 

IF  (BIFFBU  ,GE,  IJCI... )  GO  TO  3? 
GO  TO  38 

37  U.IR  I TE  (  6  *  39  )  BU 

39  FORM  AT  (*  %*BU==  "  ?  F 1 1 1 8  ) 

M-  <  BU-BL  )  /  ♦  005+1 
DO  40  K  "•  1 « N 
SL.VAL--99 . 0 
l.JVAL-0  >0 
B3-BL-  <•  005 
DO  04  J-l  y  M 
B3-B3+  ,  005 


COMPUTE  LOWER  LIMIT  FOR  "A"  GIVEN  PARTICULAR  VAI.  OF  "h 

AL -PARMA 
RED-0 , 005 
43  AL-AL-RED 
PARL-0 . 0 
DO  4.1.  1-1  ?N 

C  A I ..  - 1 . 0  -  A 1.  /  ( T I M  E  ( I )  *  *  B  3 ) 

SPA  I..  -  (  OBS  <  I )  -CAL  )  **2 
f  ■'  A  R  L.  •'■■■■  P  A  R  L  T  S  P  A  I.. 

41  CONTINUE 

DIFFAL-PARL-SSE 

IF  (DIFFAL  . GE *  UCL )  GO  TO  42 

GO  TO  43 

COMPUTE  VALUE  OF  SLOPE  FOR  GIVEN  11  AL  "  ?  "  B  "  »  T  (  K  ) 

4  2  S  I...  L  ••=  ( A  I...  *  B  3 )  /  ( T  ( K  )  *  *  ( B  3  F 1 . 0 )  > 

SAVE  SMALLEST  VALUE  OF  SLOPE  OVER  FULL  INTERVAL  OF  "  B 
LOWER  INTERVAL.  OF  "A"  AND  SPECIFIED  TIME?  11 T  ( K )  " 

I  F  C  SI...L  ,1  T.  HE  VAI... )  GO  TO  46 
GO  TO  47 
46  SLVAL  -  GLI¬ 
AL  VAI...- AL. 

BE  VAL.-B3 
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/  fill'"  PAKf'.A 
lit’  h!1»  hi  if!"  t- 1  ■ 

1-  nr;1  !■■  0  .  0 
i'll'i  4P  I  ;i  «  N 

;i  ,  (i-r-i!,l/  (  T  1,  HE  ( 1 )  * &B3  > 

PF  Ail--  ■  Or  b  t  I  >  -C  AO  '  -1 1 2 
!'  i  ■*  l.l A  !'■•!  1 1 1-  b  r  A  l ‘ 

4b  CONTINUE 

5  i  1.  F  I’ m  !.  t — p  A  RU-  s  y  t 

IF'  (DIF-TAU  *  GE  *  UCL. )  GO  TO  4 ■> 

GO  10  50 

OOMF'UT  E  0  A  LUFF  Or  SLOPE  FOR  GIVEN  ''  AiJ "  *  ”  B "  *  "  T  <  K )  " 

•••;  9  ill... ~  i  A l ! :s:  A 3  )  /  t  7  ( K  f  V *  <  B ?■  1 1 . 0 )  ) 

8  A U E  I .  A R G E 0 7  V  A !.  U E  0 F  S L 0 P  E  0 V E R  I N T E R V  A 1...  "  B " » UP P E R 
INTERVAL  OF  “A"*  AGO  SPECIFIED  TIME*  “1(10" 

IF  ( SOL  .01.  DUAL)  GO  (0  53 
AG  TO  54 

53  IJV’AL.-'SUi... 

A  GOAL  -  AIJ 
DUVAL - B3 

54  CONI  I HUE 

PRINT  OUT  UPPER  AND  LOWER  CONFIDENCE  INTERVAL  VALUES 
F 0 P  S L. 0 P E  •  G I  UR N  i  I  h E  *  *  I  <  K  )  11  A L 0 N G  U I T I  I  T HE  CO R - 
RESPONDING  VALUES  OF  * A “ •  AND  * B " 

WR I  IF  i.  6  *  55  >  I  t  T  ( K  <  *  SL  VAu  r  ALVAL  *  BLVAL  >•  UVAL  r  AlJVAl.  r  BUVAL. 
|: : t::,  FORMAT  ( •  "  *  "  1  (  "  » 1 2  ,  F4  ♦  1  *  4X  t  *  SL  VAL«  "  » F 1 0  ♦  8 » 


^  r»^f  -  --  -Lt' 


LrrJ"itfcWr-—  •  - 


1^2 
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